Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
Authors: Hyeong-Seok Choi
Juheon Lee
Wansoo Kim
Jie Hwan Lee
Hoon Heo
Kyogu Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: speech analysis
speech synthesis
voice conversion
self-supervised
information perturbation

Abstract: We present a neural analysis and synthesis (NANSY) framework that can manipulate the voice, pitch, and speed of an arbitrary speech signal.  Most of the previous works have focused on using information bottleneck to disentangle analysis features for controllable synthesis, which usually results in poor reconstruction quality. We address this issue by proposing a novel training strategy based on information perturbation. The idea is to perturb information in the original input signal (e.g., formant, pitch, and frequency response), thereby letting synthesis networks selectively take essential attributes to reconstruct the input signal. Because NANSY does not need any bottleneck structures, it enjoys both high reconstruction quality and controllability. Furthermore, NANSY does not require any labels associated with speech data such as text and speaker information, but rather uses a new set of analysis features, i.e., wav2vec feature and newly proposed pitch feature, Yingram, which allows for fully self-supervised training. Taking advantage of fully self-supervised training, NANSY can be easily extended to a multilingual setting by simply training it with a multilingual dataset. The experiments show that NANSY can achieve significant improvement in performance in several applications such as zero-shot voice conversion, pitch shift, and time-scale modification.

Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
Hyeong-Seok Choil
Juheon Leeld
Wansoo Kim
Jie Hwan Lee
Hoon Heo*
Kyogu Lee
L.e1.2,3,4 22,,44
MARG, Department of Intelligence and Information, Scoul National University SSueeroone Inc.
GSAA
3 AAIIS
fkekepa15, juheon2, wansookim, kglee) ac..una.c..... kc.kr, fviswisbus, hoon)aaupertone... .i
Abstract
We present a neural analysis and synthesis (NANSY) framework that can manip- ulate voice, pitch, and speed of arbitrary speech signal. Most of the previous works have focused on using information bottleneck to disentangle analysis fea- tures for controllable synthesis, which usually results in poor reconstruction quality. We address this issue by proposing novel training strategy based on information perturbation. The idea is to perturb information in the original input signal (e.g., formant, pitch, and frequency response), thereby letting synthesis networks selec- tively take essential attributes to reconstruct the input signal. Because NANSY does not need any bottleneck structures, it enjoys both high reconstruction quality and controllability. Furthermore, NANSY does not require any labels associated with speech data such as text and speaker information, but rather uses analysis features, i.c., wav2vee feature and newly proposed pitch feature. Yingram, which allows for fully self-supervised training. Taking advantage of fully self- supervised training, NANSY can be easily extended to multilingual setting by simply training it with multilingual dataset. The experiments show that NANSY can achieve significant improvement in performance in several applications such zero-shot voice conversion, pitch shift, and time-scale modification
set of
Introduction
Analyzing and synthesizing arbitrary speech signal is inarguably a significant research topic that has been studied for decades. Traditionally, this has been studied in the digital signal processing (DSP) field using fundamental methods such sinusoidal modeling
linear predictive coding (LPC), and it is the analysis and synthesis framework that lies at the heart of those fundamental methods 26 3 These traditional methods, however, limited in terms of controllability because the decomposed representations are still low-level representations. It is obvious that the closer we decompose a signal into high-leeel/interpeeaabe representations, the more we gain access to the controllability. Given this consideration, we aim to design a neural analysis and synthesis (NANSY) framework by decomposing a speech signal into analysis features that represent pronunciation, timbre, pitch, and energy. The decomposed representations can be manipulated and re-synthesized, enabling users to manipulate speech signals in various ways.
is worth noting that many similar ideas have been recently proposed in the context of voice conversion applications. We categorize the previous works in two ways, i.e., 1. Text-based approach,
'uddo samples: https//tinyurll.com/e/x///
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2. Information bottleneck approach. The first approach exploits the fact that the text modality is inherently disentangled from the speaker identity. One of the most popular text-based approaches is to use pre-trained automatic speech recognition (ASR) network to extract a phonetic posteriogram (PPG) and use it as a linguistic feature [41]. Then, combining the PPG with the target speaker information, the features are re-synthesized to speech signal. Another alternative approach is to directly use text scripts by aligning it to a paired source signal B3] . Although these ideas have shown promising results, it is important to note that these approaches have common problems. First, in order to extract the PPG features, it is required to train an ASR network in supervised manner, which demands lot of paired text and waveform datasets. Additionally, the language dependency of the ASR network limits the model's 's capability to be extended to multilingual settings or languages with low-resources. To address these concerns, efforts have been made to divert from using the text information and the most popular approach is to use an information bottleneck. The key idea is to restrict the information flow by reducing time/channel dimension, and crrmaliziin/quaatizziiin intermediate representations RG TO, [5.) Although these ideas have been explored in many ways, one critical problem is that there exists inevitable trade-off between the degree of disentanglement and the reconstruction quality. In other words, there is a trade-off between speaker similarity and the preservation of original content such as linguistic and pitch information.
To avoid the major concern of the text-based approach, we suggest to use two analysis features which wav2vec and newly proposed feature, Yingram. In order to preserve the linguistic information without any text information, we utilize wav2vec 2.0 [ trained on 53 languages in total [.. While the features from wav2vec 2.0 have mostly been used for downstream task, seek the possibility of using them for an upstream//genrran represent and control pitch information. Although it is the fundamental frequency (fo) that is mostly used to represent the pitch information, fo is sometimes ill-defined when there exists sub-harmonics in the signal (e.g., vocal fry) [6 II 2]. We address this issue by proposing a controllable but more abstract feature than fo that still includes information such as sub-harmonics. Because the proposed feature is heavily inspired by the famous Yin algorithm
task. In addition, we propose a new feature that can effectively
[
we refer to this feature Yingram.
Although the analysis features above have enough information to reconstruct the original speech signal, we have found that the information in the proposed analysis features share common information such as pitch and timbre. To disentangle the common information so each feature can control a specific attribute for its desired purpose (e.g.. wav2vec - linguistic information only, Yingram -> pitch information only), we propose an information perturbation approach, a simple yet effective solution to this problem. The idea is to simply perturb all the information we do not want control from the input features, thereby training the neural network to not extract the undesirable attributes from the features. Through this way, the model longer suffers from the unavoidable trade-off between reconstruction quality and feature disentanglement, unlike the information bottleneck approach.
Lastly, we would like to deal with unseen languages at test time. To this end, propose new test-time self self-adaptation (TSA) strategy. The proposed self- adaptation strategy does not tne-tune the model parameters but only the input linguistic feature, which consequently modifies the mispro- nounced parts of the reconstructed sample. Because the proposed TSA requires only single sample test-time, it adds large flexibility for the model to be used in many scenarios (e.g., low-resource language).
The contributions of this paper are as follows:
We propose a neural analysis and synthesis (NANSY) framework that can be trained in a fully self-supervised manner (no text, speaker information needed). The proposed method is based on a new set of analysis features and information perturbation.
The proposed model can be used for various applications, including zero-shot voice conver- sion, formant preserving pitch shift, and time-scale modification.
We propose a new test-time self-adaptation (TSA) technique than can be used even on unseen languages using only single test-time speech sample.
2

---

2 NANSY: Neural Analysis and Synthesis
Perturb
Analysis
Synthesis
P.S.
Filter Generator
w2v 12
w2v 1
S
spk
wav E.
E
Vocoder
wav L.S.
Energy
P.5
Source
g
Yingram
Generator
Figure 1: The overview of the training procedure and information flow of the proposed neural analysis and synthesis (NANSY) framework. The waveform is first perturbed using functions f and g. f perturbs formant, pitch, and frequency response. 9 perturbs formant and frequency response while preserving pitch. w2v denotes wav2vee encoder and spk denotes a speaker embedding network. L, P. S. E denotes Linguistic, Pitch, Speaker, and Energy information, respectively. The tilde symbol attached when the information perturbed using the perturbation functions. The dashed boxes denote the modules that are being trained.
2.1 Analysis Features
Linguistic To reconstruct an intelligible speech signal, it is crucial to extract rich linguistic infor- mation from the speech signal. To this end, resort to XLSR-53: wav2vec 2.0 model pre- traitri on 56k hours of speech in 53 languages [[]] The extracted features from XLSR-53 have shown superior performance downstream tasks such ASR, especially on low-resource languages. We conjecture, therefore, that the extracted features from this model can provide language-agnostic linguistic information. Now the question is, from which layer should the features be extracted? Recently, has been reported that the representation from different layers of wav2vec 2.0 exhibit different characteristics. Especially, Shah al. 3 showed that it is the output from the middle layer that has the most relevant characteristics to pronunciation In light of this empirical observation, we decided use the intermediate features of XLSR-53. More specifically, used the output from the 12th layer of the 24-layer transformer encoder.
Speaker Perhaps the most common approach to extract speaker embeddings is to first train speaker recognition network in supervised manner and then reuse the network for the generation task, assuming that the speaker embedding from the trained network can represent the characteristics of unseen speakers [2] (3)] Here we would like to take one step further and assume that we do not have speaker labels train speaker recognition network in supervised manner. To mitigate this disadvantage, we again use the representation from XLSR-53, which makes the proposed method fully self-supervised. To determine which layer of XLSR -3 to extract the representation from, we first analyzed the features from each layer. Specifically, we averaged the representation of each layer along the time-axis and visualized utterances of 20 randomly selected speakers from the VCTK dataset using TSNE [4 46]. In Fig. 2 we can observe that the representation from the 1st layer of XLSR-53 already forms clusters for each speaker, while the latter layers (especially the last layer) tend to lack them. Note that this is in accordance with the previous observation in [8] Taking this into consideration, we train a speaker embedding network that uses the 1st layer of XLSR-53 as an input. For the speaker embedding network, we borrow the neural architecture from ofte-offthhearl speaker recognition network [4], which is based on ID-convolutional neural networks (ID-CNN) with an attentive statistics pooling layer. The speaker embedding Lz-normalized before conditioning. The speaker embeddings of seen and unseen speakers during training are also shown in Fig. 2
Pitch Due to the irregular periodicity of the glottal pulse, we often hear creaky voice in speech, which is usually manifested as jitter or sub-harmonics in signals. This makes hard for fo trackers to estimate fo because the fu itself is not well defined in such cases [16 2]. We take a hint from the
We failed to train the model when used the wav2vec features from the last three layers.
3

---

12"h leyn
been ngg
embeddingg
Figure 2: The visualization of intermediate representations of XI.SR-53 using TSNE.
popular Yin algorithm address this issue. The Yin algorithm the cumulative mean normalized difference function (r) to extract frame-wise features from a raw waveform, which is defined as follows,
11.
if 0
d((t) (f(r)//_-,166)) deG), otherwise.
The (T) is a difference function that outputs a small value when there exists a periodicity on time-lag and it is defined as follows,
W
de(T)
Σ (x) - (1 r2( (0+) 2r,(r),
where i, r, W, and r, denote frame index, time lag, window size, and the auto-correlation function, respectively. After some post processing steps, the Yin algorithm selects fo from multiple fo can- didates. See [I3] for more details. Rather than explicitly selecting fo. we would like to train the network to generate pitch harmonics from the output of the function (T). However, (r) itself is limited to be used as a pitch feature because it lacks controllability, unlike fo. Therefore, we propose Yingram by converting the time-lag axis the midi-scale axis as follows.
Ym(m) = d<([c(m)]) - d((l((m))) (c(m) - [c(m)]) + d ((emm))) [c(m)] [e(m)]
(3)
c(m)
8r
(4)
440 2
where m, c(m), and denote midi note, midi-to-lag conversion function, and sampling rate, spectively. We set 20 bins of Yingram to represent semitone range. In addition, we set Yingram to represent the frequency between 10.77 hz and 1000.40 hz by setting W 2048 and the range of between 22 and 2047. In the training stage, the input to the synthesis network is the frequency range between 25. hz and 430.19 hz, which shown as scope in Fig. 3 After the training is finished, we can change the pitch by shifting the scope. That is, in the inference stage, one could simply change the pitch of the speech signal by shifting the scope. For example, we move the scope down 20 bins, the pitch can be raised by semitone.
midi/hz
Yingram
mel spectrogram
83/1000
68:430
scape
19/26
4/10
Figure 3: The visualization of Yingram and the corresponding mel spectrogram.
Energy For the energy feature, we simply took an average from a log-mel spectrogram along the frequency axis.

---

2.2 Synthesis Network
Fiter (Spectral envelope)
Source (Pitch harmonics)
Cenerated mel spedtrogram
Figure 4: The outputs of Gs and Gr. The two outputs from each generator are summed to reconstruct mel spectrogram.
is well-known that speech production can be explained by source-filter theory. Inspired by this, we separate synthesis networks into two parts, source generator G5 and filter generator Gr. While the energy and speaker features are common inputs for both generators, 9s and 9r differ in that they take Yingram and wav2vec features, respectively. Because the acoustic feature can be interpreted as sum of source and filter in the log magnitude domain, we incorporate inductive bias the model by summing the outputs from each generator similarly to 25 As will be discussed in more detail in the next section, even though the training loss is only defined using mel spectrograms, the network learns to separately generate the spectral envelope and pitch harmonies shown in Fig. 4 Note that this separation not only provides the interpretability to the model but also enables formant preserving pitch shifting. To summarize, the acoustic feature, mel spectrogram M, is generated as follows,
M Gs (Yingram, S, E) + GF(wav2vec, S, E),
(5)
where S and E denote speaker embedding and energy features. We used stacks of ID-CNN layers with gated linear units (GLU) [12 for generators. The detailed neural architecture of the generator described in Appendix Note that each generator shares the same neural architecture. The only difference is the input features to the networks. Finally, the generated mel spectrogram is converted to waveform using the pre-trained HiFi-GAN vocoder [4
Training
3.1 Information Perturbation
In our initial experiments, a neural network can be easily trained reconstruct mel spectrograms using only the wav2vec feature. This implies that the wav2vec feature contains not only rich linguistic information but also information related to pitch and speaker. For that reason, we would like to train GF selectively extract only the reguistic-related information from the wav2vec feature, not pitch and speaker information. In addition, we would like to train Gg to selectively extract only the pitch-related information from the Yingram feature, not speaker information. To this end, we propose to perturb the information included in input waveform x by using three functions that are 1. formant shifting (fs)), 2. pitch randomization (pr), and 3. random frequency shaping using parametric equalizer (pegl F We applied a function ff on the wav2vec input, which is chain of all three functions (ppp(pp))( On the Yingram side, we applied function g. which is chain of two functions Iss and peq so that fo information is still preserved as follows, g(x) = (s)pq(()) This way, we expect GF to take only the linguistic-related information from the wav2vec feature, and Gs to take only pitch-related from the Yingram feature. Since the wav2vec and Yingram features can no longer provide the speaker-related information, the control of speaker information becomes uniquely dependent on the speaker embedding. The overview of the information flow is shown in Fig. I The hyperparameters of the perturbation functions are described more Appendix
as follows, (xx
3.2 Training Loss
We used LI loss between the generated mel spectrogram . and ground truth mel spectrogram to train the generators and speaker embedding network. However, it is well-known that the speech synthesis networks trained with L1 or L2 loss suffer from smottsenoss of the generated acoustic feature, which results in poor quality of the speech signal. Therefore, in addition to the LI loss, we used the recent speaker conditional generative adversarial training method to mitigate this issue g Writing the discriminator as D(M, C,, c_) a (h((M, c_, c))), Choi et al. g proposed to
We used Parselmouth for fs and pr 20 PEQ implemented following 53
5

---

use projection conditioning 27 not only with the positive pairs but also with the negative pairs as follows.
h(M, ((((M)) + o
(6)
where M denotes mel spectrogram, (r)) denotes sigmoid function, e+ denotes speaker bedding from a positively paired input speech sample, and c. denotes a speaker embedding from randomly sampled speech utterance. <6-) denotes an output from the intermediate layer of dis- criminator and ()(+) denotes a function that maps input vector to a scalar value. The detailed neural architecture of DD is shown in Appendix The loss functions for discriminator Lo and generator Lg are as follows:
Lg E (M.cy (M.oy...))Pssee )opaata M-Paen (og(o C4, c_))) (oM, JM M1. Lo -E [M. Paace--)aaaaaaaap Mr-poen [og(o (og(a(((MM, (-)) Log(( c..
3.3 Test-time Self-Adaptation
Although the synthesis network can reconstruct an intelligible speech from the wav2vec feature in most cases, we observed that the network sometimes outputs speech signals with wrong pronunciation, especially when tested on unseen languages. To alleviate this problem, propose to modify only the input representation, that is, the wav2vec feature, without having train the whole network again from scratch. As shown in Fig- first compute I.1 loss between the generated mel spectrogram M and ground truth mel spectrogram M in the test-time. Then, we update only the parameterized wav2vec feature using the backpropagation signal from the loss. Note that the loss gradient (shown in red) is backpropagated only through the filter generator. Because this test-time training scheme requires only single test-time sample and updates the input parameters by targeting the test-time sample itself, we call it test-time self-adaptation (TSA).
Anahdie
Updato
Syrchesss
way
way 12
Geberater
other features
Source
M
Generater
Figure 5: The illustration of TSA.
Experiments
4.1 Implementation Details
Dataset To train NANSY on English, used two datasets, i.e., 1. VCTK 47
2. train-clean-3e
subset of LibriTTS [. We trained the model using 90% of samples for each speaker. The speakers of train-clean- 360 were included to the training set only when the total length of speech samples exceeds 15 minutes. To test on English speech samples we used two datasets; 1. For the seen speaker test used 10% unseen utterances of VCTK. 2. For the unseen speaker test used test-clean subset of LibriTTS.
To train NANSY multi-language, we used CSS103 dataset 32
CSS10 includes 10 speakers and each speaker use different language. Note that there no English speaking speaker included in CSS10. To train the model, we used 90% of samples for each speaker. To test on multilingual speech samples, we used the rest 10% unseen utterances of CSSI0.
Training We used 22,050 hz sampling rate for every analysis feature except for wav2vec input that takes waveform with the sampling rate of 16,000 hz. We used 80 bands for mel spectrogram, where FFT, window, and hop size were set to 1024, 1024, and 256, respectively. The samples were randomly cropped approximately to 1.47-second, which results in 128 mel spectrogram frames. The networks were trained using Adam optimizer with 31 0.5 and 32 0.9. The learning rate was fixed to 10-4, We trained every model using one RTX 3090 with batch size 32. The training was done after 50 epochs.
'The licenses of the used datasets are as follows: 1. VCTK - Open Data Commons Attribution License 1.0, 2. LibriTTS Creative Commons Attribution 4.0. and CSS10 Apache License 2.0.

---

4.2 Reconstruction
For the reconstruction (analysis and synthesis) tests, we report character error rate (CER (%)), and 5-scale mean opinon score (MOS ([1-5])). 5-scale degradation mean opinion score (DMOS ([1-5])). For MOS, higher is better. For DMOS and CER, lower better. To estimate the characters from speech samples, we used google cloud ASR API. For MOS and DMOS, we used amazon mechanical turk (MTurk). The details of MOS and DMOS are shown in Appendix D
Yingram fo We compared two models trained with Yingram and fo check which pitch feature shows more robust reconstruction performance. We used RAPT algorithm for fo estimation [42], which is known works sufficiently well in most cases, we first manually listened to the reconstructed samples using the model trained with fo- We first chose 30 reconstructed samples in the testset that failed faithfully reconstruct the original samples using the model trained with fo- After that we reconstructed the same 30 samples using the model trained with Yingram. Finally, we conducted ABX test to ask participants which of the two samples (A and B) sounds closer to the original sample (X). The ABX test was conducted on MTurk. The results showed that the participants chose Yingram with a chance of 68.3%. This shows that Yingram can be used as more robust pitch feature than fo, when fo cannot be accurately estimated.
reliable tracker among many other algorithms [22]. Because RAPT algorithm
Reconstruction test We randomly sampled 50 speech samples from VCTK (seen speaker) and sampled another 50 speech samples from test-clean subset of LibriTTS (unseen speaker) to test the reconstruction performance of NANSY trained English datasets. The results in Table @ shows that NANSY can perform high quality analysis and synthesis task. In addition, to test if the proposed framework can cover various languages, we trained and tested it with the multilingual dataset, CSS10 10. We randomly sampled 100 speech samples from CSS10 to test the reconstruction performance of NANSY trained on multi-language The results are shown Table 2 Although we do not impose any explicit labels for each language, the model was able to reconstruct various languages with high quality. The results of CER on each language shown in Fig. MUL.
CER
MOS
DMOS
CER
MOS
DMOS
GT
n/a
4.28 = 0.09
n/a 1.93 0.09
GT
n/a
4.19 = 0.08
n/a
Recon
5.6
4.18 = 0.09
Recon
7.3
4.14 0.09
1.74 0.09
Table 1: English reconstruction results.
Table 2: Multilingual reconstruction results.
Test-time self-adaptation To test the proposed test-time self-adaptation (TSA), compared the CER performance of NANSY trained three different configurations, 1. English (ENG), English with TSA (ENG-TSA). 3. Multi-language (MUL). For every experiment, we iteratively updated the wav2vec feature 100 times. The results are shown in Fig. X Naturally, MUL generally showed better CER performance than other configurations. Interestingly, however, ENG-TSA sometimes showed similar or even better performance than MUL, which shows the effectiveness of the proposed TSA technique.
15
ENG
10
ENG-TA
CER
MUL
HU
FR
CH
GR
DE
Figure 6: The CER results 10 languages (NL: Dutch, HU: Hungarian, FR: French, JP: Japanese,
CH: Chinese, RU: Russian, Fl: Finnish, GR: Greek, ES: Spanish, DE: German).
4.3 Voice conversion
NANSY can perform zero-shot voice conversion by simply passing the desired target speech utterance to the speaker embedding network. We first compared the English voice conversion performance of NANSY with recently proposed zero-shot voice conversion models. Next, we tested the multilingual voice conversion performance. Finally, we tested unseen language voice conversion for both seen speaker and unseen speaker targets. Note that in every voice conversion experiment, we shifted the
7

---

median pitch of a source utterance to the median pitch of a target utterance by shifting the scope of Yingram. We measured naturalness with 5-scale mean opinion score (MOS [1-5]). Speaker similarity (SSIM (%) were measured with binary decision and uncertainty options, following [50] For MOS and SSIM, we again used MTurk. The details of MOS and SSIM are shown in Appendix D One of the crucial criteria for evaluating the quality of the converted samples is to check the intelligibility of them. Previous zero-shot voice conversion models, however, have only reported MOS or SSIM and have been negligent on assessing the intelligibilityy of the converted samples 36 I0 MTI To this end, we report character error rate (CER (%) between estimated characters of source and converted pairs. To estimate the characters from speech samples, we used google cloud ASR API.
Algorithm comparison Here, we report three source-to-target speaker conversion settings, 1. seen-to-seen (many-to-many. M2M), 2. unseen-to-seen (any-to-many, A2M), 3. unseen-to-unseen (any-to-any, A2A). For every setting, we considered male (m2m), male-to-female (m2f), female-to-mabe (f2m), and Gemale-to-female (f2f). For M2M setting, we randomly sampled 25 seen speakers from VCTK and randomly assigned 2 random speakers from VCTK, resulting in 200 (=25x2x44 conversion pairs in total. For A2M setting, we randomly sampled seen speakers from test-clean subsel of LibriTTS and randomly assigned random speakers from VCTK resulting in 80 (=1022x4)) conversion pairs in total. For A2A setting, we randomly sampled 10 seen speakers from text-clean subset of LibriTTS and randomly assigned random speakers from test-clean subset of LibriTTS resulting in 80 (=10x2x4) conversion pairs in total. We trained three baseline models with official implementations VQVC+ [5]] AdaIN [TO] AUTOVC [3 using the same dataset and mel spectrogram configuration NANSY. For fair comparison, we used a pre-trained HiFi-GAN vocoder for every model. Table B shows that NANSY significantly outperforms previous models in terms of every evaluation measure. This implies that the proposed information perturbation approach does not suffer from the trade-off between CER and SSIM unlike information bottleneck approaches. The SSIM results for all possible gender-to-gender combinations are shown in Fig. 9 in Appendix
combination, i.e., male-to-
M2M
A2M
A2A
[ER[[] MOS[I-5] SSIM[%] CER[S) [6]
MOS[I-5] SSIM[%] CER[$] MOS[I-5] SSIM[%]
SRC TGT
n/a
4.23 = 0.55
0
n/a
4.28 +
0.60
n/a
4.26 + 0.07
0.25
TGT as TGT
n/a
4.32 : 0.05
94.9
n/a
4.29 1 0.05
92.4
n/a
4.27 1 0.07
96.2
VQVC+
54.0
1.76 = 0.05
54.5
74.7
1.73
- 0.11
15.6
69.3
1.83 + 0.09
13.8
AdaIN
62.9
2.22 = 0:07
24.0
79.6
1.92
0.12
18.1
59.3
2.12 - 0.10
21.2
AUTOVC
31.7
3.41 = 0:06
47.3
36.1
2.74 0.11
33.2
28.2
2.59 + 0.08
23.3
NANSY
7.5
3.79 0.07
91.4
7.6
3.73
0.05
88.1
8.6
3.44
0.07
64.6
Table 3: Evaluation results on English voice conversion. SRC and TGT denote, source and target, respectively.
Multilingual voice conversion We tested multilingual voice conversion performance with the model trained on the multilingual dataset, CSS10. We randomly sampled 50 samples for each language speaker from CSS10 and assigned random single target speaker from CSS10 for each source language, resulting in 500 (=50x 10) conversion pairs in total. The results in Table 4 show that the proposed framework can successfully perform multilingual voice conversion by training it with the multilingual dataset. However, there is still a room for improvement for multilingual voice conversion when comparing to the results in Table 5 where NANSY is just trained on English.
speaer
Unseen language voice conversion We tested the voice conversion performance on unseen source languages using NANSY trained on English. We tested the performance on two settings, unseen source language (CSS10) to seen target voice (VCTK) and 2. unseen source language (CSS 10) to unseen target voice (CSS10). For the first experiment, we randomly sampled 50 samples for each unseen language speaker from CSS10 and assigned random English target speakers from VCTK, resulting in 500 (=50x 10) conversion pairs in total. The second experiment was conducted identically to the multilingual voice conversion experiment setting. The results in Table shows that NANSY can successfully extended even to unseen language sources, although there was a decrease on SSIM compared to the results in Table (69.5% + 61.0%).
8

---

Seen Speaker
Unseen Speaker
CER
MOS
SSIM
CER
MOS
SSIM
CER
MOS
SSIM
TGT as TGT n/a
98.0 4.23 0.06
TGT as TGT
n/a
4.24 # ou7
100 n/a 4.23 ous
92.0
NANSY
18.8 3.68
69.5
NANSY
14.8 3.75 + 0.10 90.0
15.6 3.76 + ous 61.0
Table 4: The multilingual voice conver- sion results. The model was trained us- ing only CSS10.
Table 5: The voice conversion results on unseen language dataset, CSS10. The model trained using only the English datasets.
4.4 Pitch shift and time-scale modification
To check the robustness of pitch shift (PS) and time-scale modification (TSM) performance of NANSY, we compared it with other robust algorithms, i.e., PSOLA 5 and WORLD vocoder 29 28
Pitch shift We tested PS performance with 5 semitone ranges, -6, -3, 0, 3, 6. The pitch was changed by shifting the scope of the proposed Yingram feature. Note that '0" was used check analysis- synthesis performance. We randomly selected 20 samples for each semitone range from 10% unseen utterances of VCTK. We evaluated the naturalness of speech samples with MOS on MTurk. The results in Table O show that NANSY generally outperforms algorithms such as PSOLA and WORLD vocoder on PS task.
Time-scale modification We tested TSM performance with 5 time-scale ratios, 1/2, 1/1.5, 1, 1.5, 2. The time-scale was modified by simply manipulating the hop length of the analysis features. Note that '1' was used check analysis-synthesis performance. We randomly selected 20 samples for each time-scale ratio from 10% unseen ulterances of VCTK. We evaluated the naturalness of speech samples with MOS MTurk. The results in Table 7 show that NANSY achieved competitive performance on TSM compared to the well-established PSOLA and WORLD vocoder.
-6
-3
0
3
1/2
1/1.5
1
1.5
2
WORLD
3.43
3.53
3.85
3.63
3.53
WORLD 2%
2.40
3.60
3.83
3.38
3.03
PSOLA 20
3.63
3.55
3.93
3.75
3.48
PSOLA P9
2.55
3.66
3.95
3.68
2.85
NANSY
3.60
3.68
4.05
3.90
3.78
NANSY
2.45
3.63
3.98
3.70
2.93
Table 6: Pitch shift results.
Table 7: Time-scale modification results.
5
Related Works
Self-supervised representation learning and synthesis of speech There has been an increasing interest in the self-supervised learning methods within the machine learning and speech processing community, Oord et al. [3T first proposed to use noise contrastive estimation loss to train speech representations. Baevski et al. 4 extended this idea by integrating masked language modeling [15]. Another popular self-supervised learning method for speech representation is to train a neural network Most recently,
by targeting multiple self caff-uueervsiion tasks R4 3
B55 used the discrete disentangled
self-supervised representations to re-synthesize them into a waveform. Although using the discrete units has its own advantage in that it is disentangled with speaker information, we found that an inaccurate quantization process often leads to mispronounced samples, which is why we turned to continuous representation as it can provide more accurate results on linguistic information.
Zero-shot voice conversion Research on zero-shot voice conversion has been most actively con- ducted through the information bottleneck approach. Qian et al. 3o proposed to perform zero-shot voice conversion by utilizing the pre-trained speaker recognition network and information bottleneck by carefully designing the bottleneck of auto-encoder. Inspired by the success of style conversion in computer vision, Chou and Lee [[C also focused on restricting the information flow using instance normalization
[ and adaptive instance normalization [19] Lastly, Wu et al. [11 used multiple vector quantization layers 60 to restrict the information flow.
We used Parselmouth for PSOLA
20

---

Inductive bias for audio generation It has been shown that neural networks can be combined with traditional speech/sound production models for efficient and strong performance. One of the speech production models that has been integrated with neural networks is the source-filte firtee model. By modeling source and filter components with deep networks, it has been used for applications such as vocoder /9
2 45 and acoustic feature generation [25] Furthermore, Engel et al. [17 proposed
to integrate a harmonic plus noise model 38 and neural networks to produce natural audio signal.
Consistency learning Learning representations by augmenting the data has been one of the key ideas to leverage the performance of classification tasks 43 [52] This shares the similar idea with the proposed information perturbation strategy in that the data is perturbed so that the neural network must learn to ignore the perturbations and learn the consistency from the data. However, the key difference between the consistency learning and the information perturbation is that the information perturbation method is designed for "generative" task and that it is the "decoder" (e.g., Generator) that trained to selectively take the essential attributes to reconstruct the signal from the given perturbed representations.
6 Conclusions and Discussion
In this work, we proposed neural analysis and synthesis framework (NANSY) that can perform zero-shot voice conversion, formant preserving pitch shift, and time-scale modification with single model. The proposed model can be trained in fully self-supervieed manner, that is, it can be trained without any labeled data such as text speaker information. We showed the effectiveness of the proposed information perturbation approach by showing the voice conversion results on various settings. Furthermore,
showed the effectiveness of the proposed TSA method by testing it on unseen languages, which shows the possibility of NANSY to be extended on low-resource languages. Although the proposed method empowers controllability over several attributes of a speech signal, it is still limited in terms of lacking controllability over linguistic information. As a future work, therefore, would like to investigate on hybrid approach that integrates text information that the user can manipulate even the linguistic information in the speech signal. Finally, to prevent the proposed framework being used maliciously (e.g., voice phishing). would be important develop a detection algorithm that can discriminate a synthesized speech sample from a real speech sample. To examine the potential of such detection system, we have tried using the trained Discriminator from the NANSY framework, which is expected to discriminate fake samples from real samples. We measured the accuracy of classifying 185 reconstructed samples and 185 ground truth samples. In addition, we measured the accuracy of classifying 560 voice conversion samples and 560 ground truth samples. The accuracy was 91.4% on the reconstruction set and 95.5% on the voice conversion set. This shows the possibility of Discriminator being used as a byproduct network to discriminate real speech samples from fake speech samples. However, we also found that Discriminator is prone to being deceived by the generated samples from other speech generative models as Discriminator was not jointly trained with those generative models. Therefore, we expect more robust synthesized speech detection algorithms to be developed in the future such as 48 40
side input so
Broader Impacts
The proposed NANSY framework shows that generative model can benefit from self-supervised representations. By choosing proper domain specific "information perturbation" " functions, we believe that one can achieve controllable generative modeling in a fully self-supervised way. The information perturbation training strategy may also be used for other modalities and facilitate self-supervised representation learning methods too. With the proposed training framework, one can manipulate various aspects of speech samples. Among the various controllabilities, it is rather obvious that the voice conversion technique can be misused and potentially harm other people. More concretely, there are possible scenarios where it is being used by random unidentified users and contributing to spreading fake news. In addition, it can raise concerns about biometric security systems based on speech. To mitigate such issues, the proposed system should not be released without a consent so that cannot be easily used by random users with malicious intentions. That being said, there is still a potential for this technology to be used by unidentified users. As a more solid solution, therefore, we believe a detection system that can discriminate between fake and real speech should be developed. The preliminary results of the detection system is reported in section
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & communications Technology Planning Evaluation (IIP)) grant funded by the Korea goVernmen((MSIT) (NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)]
References
[1] Luc Ardaillon. Synthesis and expressive transformation of singing voice. PhD thesis, 11 2017.
[2] Luc Ardaillon and Axel Roebel. Gei detection from raw speech using fully-convolutional network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6739-6743. IEEE, 2020.
[3] Bishnu S Atal and Suzanne L Hanauer. Speech analysis and synthesis by linear prediction of the speech wave. The journal of the acoustical society of America, 50(2B):637-655, 1971.
[4] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing Systems, volume 33, pages 19449-12460 Curran As- sociates, Inc., 2020. URL https:://roceediigg neurips cc/ ppppppppp///////l///// (BBt6ee....c01111112277706660000
pdf
[5] Paul Boersma and Vincent Van Heuven. Speak and unspeak with praat. Glot International, (9/10):341-3)) 1-47, 2001.
[6] Clara Borrelli, Paolo Bestagini, Fabio Antonacci, Augusto Sarti, and Stefano Tubaro. Synthetic speech detection through short-term and long-term prediction traces. EURASIP Journal on Information Security, 2021(1):1-14, 2021.
[7] Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, sheng zhao, and Tie- Liu. Adaspeech: Adaptive text to speech for custom voice. In International Conference on Learning Representations, 2021. URL https://oppenrreieer http::///ppppeeeeeeeeeeee////rro
[8] Hyeong-Seok Choi, Changdae Park, and Kyogu Lee. From inference to generation: End-to-end fully self-supervised generation of human face from speech. In International Conference on Learning Representations, 2020. URL https: htttt::://penreviiee
[9] Yeunju Choi, Youngmoon Jung, and Hoirin Kim. Neural mos prediction for synthesized speech using multi-tast learning with spoofing detection and spoofing type classification. 2021 IEEE Spoken I Language Technology Workshop (SI.7), pages 462-469. IEEE, 2021.
[10] Ju-Chieh Chou and Hung- Yi Lee. One-Shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization. In Proc. Interspeech 2019, pages 664- 668, 2019. doi: (0.21477/nterseecch.009-9 Interspeech 2019-
URL htttp://dx. doi 1rrii/oo//11.2133/11
[11] Alexis Conneau, Alexei Baevski, Ronan Collobert, AbdeIrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006. 13979, 2020.
[12] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. International conference on machine learning, pages 933-941, PMLR, 2017.
[13] Alain De Cheveigné and Hideki Kawahara. Yin, fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 19174))91771193,, 2002.
[14] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynek. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. In Proc. Interspeech 2020, pages 3830-3834, 2020. doi: 10.21447/meerspeeh.2200-2222 URI. 21437 http://..1137//meeeepeeee....... 2020-2650
ht doi

---

[15] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-trane tre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18853//1/N19911444 URL https://ww/ aclweb. org wwwgg/nntoloogy/Naa. 1423
[16] Thomas Drugman, Paavo Alku, Abeer Alwan, and Bayya Yegnanarayana. Glottal source processing: From analysis to applications. Computer Speech & Language, 28(5):111--1138 2014. ISSN 0885-2308. doi: http:////ww..oooaa..e/co///////0
03.0000..3.3.00.. URI, UtR.ptttss:///e
sciencedirect. com ott.accceeieeecccccooccc/aaaaaed
[17] Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable
digital signal processing. In International Conference on Learning Representations, 2020. URL
https::/openreviiee www.:nee//foooo hit.:eennnnntooommdddddlellremm 1dd-B1x1mattDDr
[18] Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu. Exploring wav2vee 2.0 on speaker verification and language identification. arXiv preprint crXiv:2012.06185 2020.
[19] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501-1510, 2017.
[20] Yannick Jadoul, Bill Thompson, and Bart De Boer. Introducing parselmouth: A python interface to praat. Journal of Phonetics, 71:1-15, 2018.
[21] Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang. Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang. Ignacio Lopez-Moreno, and Yonghui Wu. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In Advances in Neural Information Processing Systems, pages 4485-4495, 2018.
[22] Denis Jouvet and Yves Laprie. Performance analysis of several pitch detection algorithms simulated and real noisy speech data. In 2017 25th European Signal Processing Conference (EUSIPCO), pages 161 1614-1618. IEEE, 2017.
[23] Lauri Juvela, Bajibabu Bollepalli, Vassilis Tsiaras, and Paavo Alku. Glotnet-a wave- form model for the glottal excitation in statistical parametric speech synthesis. /EFE/ACM Transactions on Audio, Speech, and Language Processing, 27(6):001- 2019.
[24] Jungil Kong. Jachyeon Kim, and Jaekyoung Bae. Hifi- gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 77022-17033 Curran Associates, Inc., 2020. /RL.htttpss//rrrceddin.....
[25] Juheon Lee, Hycong-Seok Choi. Chang- Bin Jeon, Junghyun Koo, and Kyogu Lee. Adversarially Trained End-to-End Korean Singing Voice Synthesis System. In Proc. Interspeech 2019, pages 2588-2592, 2019. doi: 00.21497/nterspecch.211999
URL http: wwi. org/10
21437/Interspeech. 2019-1722
[26] Robert McAulay and Thomas Quatieri. Speech analysis/synthesis based on a sinusoidal representation. IEEE Transactions on Acousties, Speech, and Signal Processing, 34(4):744-754 1986.
[27] Takeru Miyato and Masanori Koyama, cGANs with projection discriminator. In International Conference on Learning Representations, 2018. URL ltttps://operreviee http:/ewwwwvnvneffffeeeeeee net/forum?
id BdBBySVVpRRZ
[28] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: a vocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 9977)1187711884 2016.
[29] Eric Moulines and Francis Charpentier. Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones. Speech communication, 9(5-6):453-46, 1990.
12

---

[30] Aaron van den Oord, Oriol Vinyals. and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arXiv:/711.00931 2017.
[31] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 807.03748, 2018.
[32] Kyubyong Park and Thomas Mulc. CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages. In Proc. Interspeech 2019, pages 1566-1570. 2019. doi: 10.21457/mmtesspeechh. 2019-1500. URL http:///ssss dw.. org/ /00 10. 21133/Interreeeeeeeee -tt.:/eeecch.111110000000 1500-1500
[33] Seung-won Park, Doo-young Kim, and Myun-chul Joe. Cotatron: Transcription-guide speech encoder for any-to-many voice conversion without parallel data. arXiv preprint arXiv:2005.03295. 2020.
[34] Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. Learn- ing Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks. In Proc. Interspeech 2019, pages 161-165, 2019. 2019-2603
doi: 10.21437/mterxpeech.2000
URL
http://dx..oe doi. org/
[35] Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech Resynthesis from Discrete Disentan- gled Self-Supervised Representations. In Pruc. Interspeech 2021, pages 3615-3619, 2021. doi: 2021444//meerspecch22211-11
[36] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pages 5210-5219. PMLR, 2019.
[37] Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6989-6993. IEEE, 2020.
[38] Xavier Serra et al. Musical sound modeling with sinusoids plus noise. Musical signal processing, pages 91-122, 1997.
[39] Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah. What all do audio transforner models hear? probing acoustic representations for language delivery and its structure. arXiv preprint arXiv:2101.00333 2021.
[40] Arun Kumar Singh and Priyanka Singh. Detection of ai-synthesized speech using cepstral & bispectral statistics. arXiv preprint arXiv:2009.00 00900490000999..... 2020.
[41] Lifa Sun, Kun Li, Hao Wang, Shiyin Kang, and Helen Meng. Phonetic posteriorgrams for many- to-one voice conversion without parallel data training. In 2016 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2016.
[42] David Talkin and W Bastiaan Kleijn. A robust algorithm for pitch tracking (rapt). Speech coding and synthesis, 495:518, 1995.
[43] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.01780. 2017.
[44] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: arXiv: 1607.08022, 2016.
[45] Jean-Marc Valin and Jan Skoglund. Lpenet: Improving neural speech synthesis through linear prediction. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and IEEE, 2019.
Signal Processing (ICASSP), pages
[46] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
13

---

[47] Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of large regional accent speech database. In 2013 international conference oriental COCOSDA held jointly with 2013 conference on Asian spoken language research and evaluation (O-C (O-COCCOSDACCALLRE)) pages 1-4. IEEE, 2013.
[48] Run Wang, Felix Juefei-Xu, Yihao Huang, Qing Guo, Xiaofei Xie, Lei Ma, and Yang Liu. Deepsonar: Towards effective and robust detection of ai-synthesized fake voices. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1207-1216, 2020.
[49] Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter-based waveform model for statistical parametric speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5916-5920, IEEE, 2019.
[50] Mirjam Wester, Zhizheng Wu, and Jumichi Yamagishi. Analysis of the voice conversion challenge 2016 evaluation results. In Interspeech, pages 1637-1641, 2016.
[51] Da-Yi Wu, Yen-Hao Chen, and Hung yi Lee. VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net Architecture. In Proc. Interspeech 2020, pages 4695,, 2020. doi: 10.214477/nterspeech..22 20200 1443. 8020-1443
www.ddo...oogggoooooo /00 21437 http://///....214////eeeeeeeee Interspeech
[52] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 1904. 12848, 2019.
[53] Vadim Zavalishin. The art of va filter design. Native Instruments, Berlin, Germany, 2012.
[54] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: corpus derived from librispeech for text-to-speech. In Interspeech, pages 1530, 2019.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We described the goal and contribution of this paper and conducted experiments accordingly.
(b) Did you describe the limitations of your work? [Yes] We explained the limitation of this work in the conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discussed the potential negative societal impacts (e.g., voice phishing) of our work in the conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [No] We do not include theoretical results.
(b) Did you include complete proofs of all theoretical results? [No] We do not include theoretical results.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as URL)? [No] The code proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section 4. and Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We have included the error bars for crowdsoureing evaluation.
14

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See section 4.1
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited all three datasets we used for experiments.
(b) Did you mention the license of the assets? (Yes] See section
(c) Did you include any new assets either in the supplemental material or as URL? [No] We did not curate/release any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you't 'r using/curating? [Yes] We cited the paper of the datasets explain all the details regarding speaker recruitment.
using in which they
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We used speaker labels split the datasets as described in section
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] We have attached the screenshots of MTurk instructions in Appendix
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Ys]] This work is approved by IRB (IRB No. 2105/004-008). We have announced that the de-identified information such as worker II will be collected through MTTurk instructions.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes] It is shown in Appendix D
15


Reviews:
[{"review_id": "tVjTWHvs8SY", "reviewer": "Reviewer_85wX", "summary": "This paper presents a framework for speech conversion using self-supervised representations. The framework, called NANSY, consists of two parts: analysis and synthesis. A wav2vec pre-trained model is used to extract speech representation to train the filter generator in the synthesis module. Similarly, the wav2vec features from the first layer is used to estimate a speaker embedding network. The speaker embeddings are used as inputs to both the source and filter generators.   A novel Yingram feature is proposed to train the source generator. Yingram is derived by mapping the difference function from the Yin algorithm (well-known for pitch estimation) from the time-lag axis to the midi-scale axis. The outputs from the analysis module can be controlled to manipulate the output speech (useful for voice conversion and time/pitch scale modifications). The paper also proposed a method for test-time self-adaptation by updating the wav2vec representation features using the L1 loss between the original and reconstructed spectrograms. \n\nThe effectiveness of the proposed framework was evaluated on several tasks using a combination of the VCTK, LibriTTS and CSS10 multilingual datasets:\n- speech reconstruction: \n  - the speech reconstructed using Yingram is preferred over f0 in a subjective A-B test.\n  - the reconstructed speech has similar MOS scores compared to the original speech.\n  - test-time self-adaptation can achieve slightly better character error rate (CER) performance when the reconstructed speech is used in a downstream ASR task.\n- Voice conversion:\n  - Compared NANSY with several existing methods (VQVC+, AddIN and AUTOVC) and showed that NANSY can achieved significantly better CER performance, naturalness MOS and speaker similarity.\n  - NANSY works for multilingual and unseen language voice conversion.\n- Pitch and time modifications:\n - NANSY achieved better pitch shift modification performance compared to WORLD and PSOLA.\n - NANSY achieved similar time scale modification performance compared to WORLD and PSOLA.\n\n ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe main novelty of the proposed NANCY framework is the use of the wav2vec speech representation to avoid having a bottleneck structure (used in existing voice conversion work to have better controllability) and therefore can achieve good quality  reconstruction. The Yingram feature is proposed to achieve better control for pitch modification. Speech perturbation is used to encourage the generators in the synthesis module to select the relevant attributes.\n\nQuality:\n\nOverall, the design of the NANSY framework is well-motivated and it appears to achieve good performance improvements for voice conversion. The results in table 3 show that NANSY is substantially better compared to other methods, especially for the CER and SSIM metrics. Please provide explanation for the large improvements observed. It is also important to clarify that NANSY uses the pre-trained wav2vec model, which has been trained large amount of speech data that were not used in the other models.\n\nClarity:\n\nOverall, the paper is well written and easy to follow. However, there are several aspects that need clarification:\n- For the Yingram vs f0 comparison in Section 5.2, it seems biased if the ABX test was done based on the failure cases using f0. It would have been better to select half of the samples based on the failure cases using Yingram.\n- For the test-time self-adaptation, it seems to assume that the ground truth spectogram is available to compute the L1 loss. How could this be useful in practice when ground truth is not available at test time.\n- For the speaker embedding network, it is unclear how it is trained without speaker labels. It is important to provide more traning details.\n \nSome minor comments:\n- Line 288: \"google\" $\\rightarrow$ \"Google\".\n- Line 311: \"there is still a room\" $\\rightarrow$ \"there is still room\".\n- In various tables, the CER results for the ground truth or target are n/a. Why are they not available?\n\nSignificance:\n\nThe proposed framework leverages on the recent advancement in speech representation learning to achieve high quality speech reconstruction and voice conversion. In particular, the improvement in voice conversion quality is quite substantial.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is lack of ablation studies to understand the importance of speech perturbation on reconstruction quality. There are also no comparison with other techniques for the results in Tables 4 and 5.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "mSdNwd_tE44", "reviewer": "Reviewer_qupx", "summary": "This paper proposes a neural framework for manipulating voice, pitch, and speed of speech signals trained in a self-supervised manner without any labels such as text or speaker information. It does so by using features from wav2vec 2.0 and their newly proposed Yingram, with data augmentation on unrelated dimensions to focus on extracting of linguistic information from wav2vec and pitch information from Yingram. The framework can perform zero-shot voice conversion, even to unseen languages, with the use of test-time self-adaptation, which updates the wav2vec feature via backprop using the reference utterance from the unseen language. Results in voice conversion, pitch shifting, and time-scale modification are impressive.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\nI am not familiar with work in this area. Using features from wav2vec for generation seems straightforward, and the idea of perturbing features to avoid the model capturing them is definitely not new. Source-filter models have also existed for some time. But I do think combining everything together in this way is an original approach.\n\n\n# Quality\nThe submission is high quality and includes comparisons against various baselines on a variety of objective and subjective metrics. There are just a few points I'm curious about.\n\n- In section 5.2, it is noted that Yingram works better in the case when f0 cannot be accurately estimated. But how does it compare in the common case where f0 estimation is decent?\n- The use of the Yingram is to disentangle pitch information and allow pitch shift. I am curious whether the voice conversion works fine with just the wav2vec feature, where the pitch is not perturbed. If voice conversion with just wav2vec works just as well, then it is arguable that the contribution of this paper is closer to \"introduce a novel Yingram feature to allow for pitch shift (and also test-time self-adaptation)\", which, while it is still a good paper, is a lot less exciting.\n- It is not clear to me whether the speaker conditional adversarial loss is necessary. The HiFi-GAN vocoder should be able to produce good samples even from an oversmoothed spectrogram.\n\n\n# Clarity\nThe submission is extremely well written. The prose is solid and sections flow well. Enough information is provided to reproduce the work.\n\n\n# Significance\nThe results, especially the demos, are impressive. The ability for zero-shot transfer of speaker voice even to unseen languages can have major impact on content dubbing. It would be more exciting if textual information can be controlled as well, as noted in the conclusions.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors note the potential negative societal impact, and hint at the need to have a method to detect such results from their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "2. Raise safety or security concerns. For example: is there a risk that applications could cause serious accidents or open security vulnerabilities when deployed in real-world environments?\n\nVoice conversion can affect speaker id based security. \n\n7. Deceive people in ways that cause harm. For example: could the approach be used to facilitate deceptive interactions that would cause harms such as theft, fraud, or harassment? Could it be used to impersonate public figures to influence political processes, or as a tool of hate speech or abuse?\n\nZero-shot voice conversion can be used to impersonate anyone.\n\n\nThe authors acknowledge these issues and hint at the need to have a method to detect such results from their work."}}, {"review_id": "i3H0M3zEa9", "reviewer": "Reviewer_C3ov", "summary": "The paper proposes a speech analysis/synthesis framework that allows users to manipulate speech traits such as pitch, identity, and time scale. The framework exploits existing self-supervised representations for speaker identity and content, and develops a new one for pitch (Yingram). Results are shown for speech reconstruction, voice conversion, pitch shift, and timescale adaptation. Emphasis is put on multilingual and zero-shot scenarios.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and shows a thorough evaluation on different but related tasks that I think do a good job in convincing the reader that their method works. The audio samples provided are impressive, which made me increase my review score. The fact that existing self-supervised representations can be used to re-synthesize speech is far from clear (and, to me, rather unintuitive), and the paper could be considered a first work in showing that.\n\nOriginality --- The paper borrows from existing work to construct a system that, although not very original, works well in terms of performance.\n\nQuality --- Given the evaluations done and the audio samples provided, my impression is that the system performs quite well. I'd have liked to see some intermediate evaluations (see below) but the whole result is ok.\n\nClarity --- The explanations are clear although they could be improved a bit (see below for some suggestions). \n\nSignificance --- I think that the paper will hardly inspire other machine learning disciplines. However, I think it sets a bar for the sub-topic of speech modeling and synthesis. The focus on multilingual and zero-shot scenarios is of relevance. There is also the aforementioned point of showing that pre-trained self-supervised representations can be exploited to synthesize speech with good results.\n\nAdditional comments:\n* I do not very much agree with the classification between \"text-based\" and \"information bottleneck\" approaches. I find that there are many approaches, specially in voice conversion (VC), that do not fall within these two categories and that, therefore, such classification is too much of a reductionist one. A couple of examples of such VC approaches that the authors could consider citing from past NeurIPS are [A,B].\n* I think it would be much clear if authors showed the losses they use in Fig 1. Do I understand correctly that, if the vocoder is pre-trained, all losses are applied before the vocoder?\n* I think more detail could be provided in the \"Pitch\" section of page 4. Specially for the proposed new feature. It could be also interesting that the authors shared their code for this part in the future, to foster further re-use.\n* In Line 171, it is mentioned that a pre-trained HiFi-GAN vocoder is used, but this somehow seems to conflict with the multi-language setting right? (was HiFi-GAN trained with multiple languages or shown to perform well in such scenario?) It would have been interesting to comment that and perhaps include some experiment to assess the \"effect\" of using this pre-trained GAN (and not another one) in their approach. Which is the impact? How much success does depend on the vocoder? ... Which would be the MOS of HiFiGAN alone in Tables 1-2? Which would be the MOS of the same system with another vocoder?\n* Similarly for the \"information perturbation\" approach. Which is the effect of not using it? Could the authors provide any quantitative measurement? In general, I find that the paper lacks some ablation or quantitative assessment of the different proposals. It would be good to provide such assessment to try to elucidate which are the crucial/critical components of the system.\n* Please do not use more than one character for mathematical variables and functions. For instance, $fs$ in mathematical notation means $f$ multiplied by $s$. Also, in that case, $f$ is also confusing with function $f$ (Sec 4.1). I consider using more than one character simply bad practice and should be avoided at all cost in papers for renowned conferences like NeurIPS.\n* For the VC case, while reading Sec 5.3, I was a bit confused on how the identity of the target speaker was extracted. Perhaps this part can be improved (it helped me quite a bit the diagrams shown in the demo/samples page).\n\n[A] https://papers.nips.cc/paper/2018/hash/4559912e7a94a9c32b09d894f2bc3c82-Abstract.html\n\n[B] https://proceedings.neurips.cc/paper/2019/hash/9426c311e76888b3b2368150cd05f362-Abstract.html\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, this is done in the conclusions section.", "review_text": "", "extra_scores": {"ethical_concerns": "I could not spot any ethical issue.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "fjMqn0JeyqB", "reviewer": "Reviewer_DLtc", "summary": "The paper attempts to separate speech into four stream of features, the phonetic content, the speaker, the energy, and the fundamental frequency. The four stream are then reassembled back into the original input speech. Some of the streams, namely the energy and the fundamental frequency, are more straightforward to extract. While the others, namely the phonetic content and the speaker, rely the use of wav2vec 2.0. The separation of information is further encouraged by adding perturbation, as an approach to corrupt the information in one stream encouraging the model to pay attention to the other streams. Experiments are evaluated based on the quality of reconstruction, the ability to do voice conversion while holding certain property of the speech fixed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I consider this as a system building paper, and give the paper a score 6. The novelty is in the combination of existing techniques, and the experiments are mainly about comparing surface performance, such as CER and MOS, and not about revealing properties of the proposed model. The presentation has room for improvement, particularly for the fragmented sections. The approach relies on many assumptions and has many limitations, and those should be made clear in the paper.\n\n# Novelty\n\nThe paper's novelty is slightly lacking, because most of the ideas, even the combination of some, have already been proposed.\n\nThe idea of separating speech into multiple (more or less) independent streams has been explored, for example in [33], as the paper openly acknowledged. The use of wav2vec features for synthesis are also studied, particularly in the ZeroSpeech challenge. The use of perturbation to control or even to shift the attention of the model on certain streams is also used in the past, again in [33] for example.\n\nThe training loss is largely borrowed from [7], and the test-time adaptation has also been studied in fine-tuning large pre-trained models.\n\nThat said, as a system building paper, the particular combination is still interesting.\n\n\n# Approach\n\nThe major assumption is that speech can actually be separated into four, almost independent, streams. This is largely false. For example, duration, a major aspect of prosody, is missing in the picture. The duration information is probably spread across the wav2vec, the energy, and the fundamental frequency streams. Another example is accent. Certain words are pronounced in certain ways depending on the speaker, so the phonetic features in wav2vec might contain the speakers' preference.\n\nThe perturbation approach takes advantage of that certain properties in speech are easy to manipulate. This is a source of prior knowledge, and the limitation is the approach can only be applied to things that we can manipulate. The perturbation approach also does not help solve the problems regarding duration and accents.\n\nThe reliance on wav2vec is reasonable, but at the same time a little concerning. The design of wav2vec is to encourage the learning of phonetic content. We do not know for certain that the model is going to learn the phonetic content, though a lot of design effort has been put in to make this happen. I am not entirely sure how much the proposed model depends on the first-layer wav2vec features being speaker specific. We do not know what promotes wav2vec to learn speaker information at the first layer, and we also do not know whether the behavior is reliable (for example, when we change the hyperparameters or the data that is trained on).\n\n\n# Presentation\n\nThe presentation has some room for improvement.\n\nThe paper in general is too fragmented, that is, having many isolated subsections and paragraphs without discussing them together. Section 2, 3, 4, 5 are particularly fragmented. It would be great to have the following.\n\n* a brief overview of the model and training in the intro\n* a paragraph in section 2 about the landscape of the related work\n* a paragraph in section 3 about the models at a high level (such as the captions in Figure 1)\n* a paragraph in section 3 about how the individual components interact with each other\n* a paragraph in section 4 about how the training interacts with the model design\n* a paragraph in section 5 about the overall goal of the experiments\n\nThe related work section comes too soon. It's hard to contrast with the prior art without even understanding what is being proposed. In fact, the related work section in the current form does not put the proposed model in context and does not make any comparison.\n\nThe figures really help improve the understanding.\n\n\n# Experiments\n\nThe experiments further solidify that this is a system building paper. All experiments are about comparing to others at the high level with surface metrics, such as CER, MOS, and speaker similarity. It is hard to draw any conclusion beyond that the system works to better than the ones compared. It would be better to have experiments that support or reject hypotheses, revealing properties of the models.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Voice conversion can be particularly concerning for its potential negative uses (such as identity theft) and for its bias on underrepresented groups. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "bGZYnf-O-z", "reviewer": "Reviewer_pDDj", "summary": "The authors propose a speech autoencoder based on the source-filter model of speech production.\n\nThey propose to use pre-trained and fixed feature extraction modules based on wav2vec2 and a newly proposed \"Yingram\" feature. Using different layers of the w2v2 network, they extract a linguistic embedding and a speaker embedding. An energy feature is computed directly from the input mel spectrogram. The Yingram feature, based on the Yin algorithm, is a pitch feature that encompasses both the fundamental frequency and its harmonics.\n\nThe synthesis network is composed of a filter and source generator which are conditioned on subsets of the above features using domain knowledge. The source and filter representation are combined together and trained with a deterministic regression loss on the target mel spectrogram, as well as a GAN-based adversarial loss to compensate for faulty independence assumptions assumptions induced by the diagonal laplace distribution implied by the L1 distance. \n\nA desired end-goal of this autoencoder is to gain control over speaker identity and pitch characteristics in the reconstruction (e.g. a voice conversion or pitch shifting task). Due to redundant information encoded by the wav2vec2 and Yingram features, the authors propose to use data augmentation in feature space to provoke invariance to speaker identity in the wav2vec2 network and invariance to linguistic and speaker properties in the Yingram feature.\n\nThe authors transform the input to wav2vec2 with a parametric equalizer, pitch randomization, and formant shifting. This encourages the encodings of the wav2vec2 output to have random fluctuations due to pitch-related phenomena, while the linguistic aspects are kept constant. They similarly transform the Yingram feature with a parametric equalizer followed by formant shifting so that the fundamental frequency is preserved. Since speaker identity is scrambled in both features, they reason the generator will get this information from the speaker embedding. \n\nThe system is self-supervised, as it can be trained with no text transcriptions or speaker annotations. \n\nThe authors present experiments on downstream tasks:\n* zero-shot voice conversion\n* reconstruction\n* pitch shifting and time-scale modification\nand present voice conversion comparisons to baselines VQVC+, AdaIN, and AutoVC.\n\nThese analysis features are motivated in contrast to traditional features such as phonetic posteriorgrams (from an ASR network) or an \"information bottleneck\" latent encoding of the audio from the perspective of:\n* data and label efficiency\n* reconstruction quality\n* language independence\n\nThe authors claim that this avoids the trade-off between reconstruction quality and disentanglement that affect prior work.\n\nFinally, the authors present a method for generalizing a pre-trained model to an unseen language.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This work is the first I've seen using a pre-trained self-supervised audio network for general analysis/synthesis and voice conversion. I don't believe the \"information perturbation\" method described is novel, but this may be its first application to speech autoencoding to provoke feature invariances.\n\nQuality and clarity: The manuscript is well written and clearly explained. The voice conversion results are very strong relative to their baselines, and it is particularly surprising how poor the baselines perform on intelligibility. However, I'm not familiar with the current voice conversion state of the art, and there are many VC models I know of that were not mentioned.\n\nSignificance: The relevance of high quality mel spectrogram analysis and synthesis using language independent methods, based on domain knowledge and inductive biases such as the source-filter model is of high relevance to speech practitioners and researchers. I imagine this work will encourage further exploration of voice conversion tasks, as well as inspiring work in text-to-speech synthesis, bandwidth extension, speech enhancement / separation, and more. \n\nSince the work is built on speech-specific intuition (such as the source-filter model), I expect it to be of relatively low significance to general machine learning practitioners, or those in other disciplines within machine learning. As such, I'm not sure that NeurIPS is the best venue for this work, and think a speech conference would be a better fit.\n\nMissing related work: The related work section is missing many references to current voice conversion methods: ConvS2S-VC, StarGAN-VC, CycleGAN-VC, VoiceGrad, and their variants, for example.\n\nLearning invariances through data augmentation is not a new idea in general, though this may be its first application to speech autoencoding. I think it may be confusing and inaccurate to claim this is a novel technique, so I suggest these claims be tempered. For example, consistency regularization methods like UDA are closely related:\n\nXie, et al. \"Unsupervised Data Augmentation for Consistency Training\" (NeurIPS 2020) \n\nQuestions and clarifications:\n* Please indicate the dimensionality/frequency of the wav2vec2, Yingram and speaker embedding features and compare the dimensionality to other methods that use information bottlenecks.\n* Please indicate the number of male/female speakers in the train-360 subset of LibriTTS you use after filtering to greater than 15 minutes.\n* Yingram vs. $f_0$: Manually selecting 30 $f_0$ losses to evaluate Yingram against is a biased method of selection and it's not clear what this is meant to demonstrate. Could you explain the motivation behind it? I am left wondering what happens if you use a random sample? Or what happens if you picked the 30 cases where $f_0$ performed the best?\n* Reconstruction test: Confusing that Table 1 and Table 2 are not split by seen vs. unseen speaker. Are these tables the combined results of both seen and unseen speakers?\n* \"Note that in every voice conversion experiment, we shifted the median pitch of a source utterance to the median pitch of a target utterance by shifting the scope of Yingram\" -- please explain the motivation of this in the manuscript, as it's not clear why this was done.\n* Pitch shift and time-scale modification results: the MOS metrics don't assess whether the desired pitch shift or TSM was applied. How precise was NANSY at controlling these effects?\n* I don't believe the impact of the adversarial losses on the performance of the model was mentioned in the text -- it would be very interesting to see an ablation of the adversarial losses.\n\nThe test-time self adaptation method was difficult for me to understand because the prose refers to the \"input parameters\" (which seems like a contradiction to me?). I was unclear whether the feature itself was being optimized or the weights of the wav2vec2 network were being fine-tuned to the target language. Is this an accurate description of how you would use this method to improve voice conversion in an unseen language?\n1. For an unseen language, source spectrogram $M_s$, you generate an $\\hat{M_s}$ using the network, compute the L1 distance.\n2. Compute the gradient of the computed wav2vec2 feature $W_s = \\textrm{wav2vec2}(M_s)$ (not the weights of the wav2vec2 network).\n3. Optimize the **feature** $W_s$ with gradient descent such that it achieves a better mel spectrogram reconstruction to get $\\hat{W_s}$.\n4. Take a target speaker spectrogram $M_t$, and compute a speaker embedding for the speaker.\n5. Run your generator network again with $\\hat{W_s}$, $\\textrm{Yingram}(M_s)$ and $\\textrm{speaker}(M_t)$ as inputs to get a mel spectrogram in the source speaker's language with the target speaker's voice.\n\nIs that correct?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As stated in my main review, I believe the work is of limited relevance to general machine learning practitioners since it is built on specific domain expertise in speech processing -- the source filter model, for example. \n\nIf instead the focus of the paper is on proposing the \"information perturbation\" idea then I would expect to see examples of the method working in other domains beyond speech (language, images, etc.) and comparison and contrast to similar methods outside of speech.\n\nThe main downside of this work in terms of societal impact is that it can enable misuse, abuse and impersonation. Research into countermeasures for generative models of speech is of crucial importance (and an active area of investigation for multiple groups in the community).", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision has been reviewed and the paper has been accepted.  The authors are encouraged to address other potential harms (e.g., bias, privacy) in their final version.

---

It came to the attention of the program chairs and ethics review chairs late in the review process that this paper concerns voice conversion, a technology that can easily be used in applications that deceive people in ways that cause harm (see https://neurips.cc/public/EthicsGuidelines). This concern was mentioned by some of the reviewers. However, the paper had not been flagged for ethics review and therefore did not go through the formal ethics review process.

This paper was subsequently discussed between the ethics review chairs and program chairs.  In the end, a decision was made to conditionally accept the paper.  Ethics reviewers were brought in to help set conditions.  The list of conditions for acceptance is as follows:

1. Meaningful broader impacts statement. Moving beyond superficial discussion of obvious harms into a more detailed and thorough reflection on ethical issues, especially possible broader impact and potential for misuse. 

2. Restricted release of model through some type of licensing or form-restricted access (ie. private repo accessed via request, model code and data use restricted by licenses, etc.)

3. Discussion of possible theoretical and practical mitigation strategies for minimizing harm of such technologies in the future. If this is not possible to discuss, include a clear articulation of the limits of such models in the absence of mitigation approaches. It is not necessary to implement the mitigation strategies discussed though some current work in this area should be highlighted by authors in the main text. 

The original meta-review from the AC follows.

---

This paper proposes a neural analysis and synthesis training framework, with a novel training strategy based on information perturbation and the help of wav2vec 2.0 and Yingram. All reviewers had good comments on the novelty and significance of the work. Detailed response also addressed the concerns in the initial review comments. I recommend to accept this paper.  
Decision: Accept (Poster)