NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis
Authors: Hyeong-Seok Choi
Jinhyeok Yang
Juheon Lee
Hyeongju Kim
Venue: ICLR 2023 poster (2023)
Keywords: voice synthesis
integrated framework
zero-shot voice conversion
text-to-speech
singing voice synthesis
voice designing

Abstract: Various applications of voice synthesis have been developed independently despite the fact that they generate “voice” as output in common. In addition, most of the voice synthesis models still require a large number of audio data paired with annotated labels (e.g., text transcription and music score) for training. To this end, we propose a unified framework of synthesizing and manipulating voice signals from analysis features, dubbed NANSY++. The backbone network of NANSY++ is trained in a self-supervised manner that does not require any annotations paired with audio. After training the backbone network, we efficiently tackle four voice applications - i.e. voice conversion, text-to-speech, singing voice synthesis, and voice designing - by partially modeling the analysis features required for each task. Extensive experiments show that the proposed framework offers competitive advantages such as controllability, data efficiency, and fast training convergence, while providing high quality synthesis. Audio samples: tinyurl.com/8tnsy3uc.

NANSY++: UNIFIED VOICE S SYNTHESIS WITH NEURAL ANALYSIS AND SYNTHESIS
Hyeong-Seok Choil.2 "Jinhyeok Yang', "Juheon Lee 'Hyeongju Kim? SSouul National University
"Supertone, Inc.,
(kekepa15, yangyangii, juheon2, by
ABSTRACT
Various applications of voice synthesis have been developed independently de- spite the fact that they generate "voice" output in common. In addition, most of the voice synthesis models still require a large number audio data paired with annotated labels (e.g., text transcription and music score) for training. To this end, we propose a unified framework of synthesizing and manipulating voice signals from analysis features, dubbed NANSY++. The backbone network of + is trained in self-supervised manner that does not require any annotations paired with audio. After training the backhone network, we efficiently tackle four voice applications i.e. voice conversion, tex--to-speech, singing voice synthesis, and voice designing- by partially modeling the analysis features required for each task. Extensive experiments show that the proposed framework offers competitive advantages such as controllability, data efficiency, and fast training convergence, while providing high quality synthesis. Audio samples:
INTRODUCTION
Most deep learning-basee voice synthesis models consist of two parts: generating mel spectro- gram from an acoustic model that takes labeled annotations as input (e.g.. text, music score, etc.), and converting the mel spectrogram into a waveform using vocoder (Wang et al. 2017 Kim al.] 2020 Jeong et al. 2021 Lee et al. 2019 Liu et al. 2022). However, this usually suffers from poor synthesis quality due to the training/infeer inference mismatch of the acoustic model and vocoder. End-to- end training methods has been recently proposed to tackle such issues (Birikowski et al.] 2020 Weiss) (et al., 2021;; Donahue et al./ 2021 Kim et al. 202I). Despite the high quality, however, the train- ing process of end-to-end models often costly, as the waveform synthesis part needs be trained again when training each different model. Furthermore, regardless of the training strategies of previ- ous studies (end-to-end
not), most of the standard voice synthesis models are not modular enough in that most of the desirable control features are entangled in a single mid-level representation, or so-called latent space. This limits the controllability of such features and restrains the possibility of models being used as co-creation tools between creators and machines. Lastly, although many voice synthesis tasks are analogous in that they are meant to synthesize or control certain attributes of voice, the methodologies developed for each application remain scattered in research fields. These problems call for the need of developing a unified voice synthesis framework.
We stick to three objectives for designing the unified synthesis framework, that is. 1. data-scalable: the training procedure should be done via a minimum amount of laheled dataset while exploiting abundant audio recordings without labels, 2. modular: the training for each application should be done in a modularized way by sharing a universal parameterized synthesizer, 3. high quality: the synthesis quality must persist high standard even by abiding by the modularized training procedure. To this end, we make a core assumption that most of the voice synthesis tasks can be defined by synthesizing and controlling four aspects of voice, that is, pitch, amplitude, linguistic, and timbre. This motivates us to develop a backbone network that can analyze voice into the four properties and then synthesize them back into waveform. On that account, we propose NANSY ++, which
*Equal contribution

---

is improved upon the previous Neural ANalysis and SYnthesis (NANSY) framework (Choi al. 2021a), by putting forward a new end-to-end self-vuper training method.
First, we propose self-supervised fundamental frequency (Fo) estimation training method that can be trained without any post processing or synthetic datasets. Next, we adopt two self-supervised training methods information perturbation and bottleneck extract linguistic representation that disentangled from other representations (Choi et al. 2021a Qian et al.) (022). Then, we propose to encode timbre information using a content-dependent time-varying speaker embedding, which successfully captures timbre information of unseen target speaker during training. Finally, we propose high quality synthesis network convert the 4 analysis representations into a waveform by adopting an inductive bias of human voice production model.
We assume that by exploiting the proposed self-supervised disentangled representation learning strategies on the backbone network, we can encourage several downstream generative tasks to be more data efficient, while not losing the modularity and synthesis quality. Therefore, after train- ing the backbone network, introduce 4 exemplar applications voice conversion, text-to-speech (TTS), singing voice synthesis (SVS), voice designing (VOD)- - that can be tackled by sharing anal- ysis representations and synthesis network of the backbone. Each application can be simplified and substituted into the task of synthesizing a subset of analysis representations. Through extensive ex- periments, verify that NANSY++ enjoys lot of advantages at once that existing methodologies cannot: high quality output, fast training of modularized application models, data efficiency, and controllability over disentangled voice features.
NANSY++ ++
NANSY++ Analyzer
NANSY Synthesizer
Piich encoder
CQT
Pitch Analysia ki
Fo
Ampiltute
Sinusold &Noise Generator
Sample level Synthasis
Linuletic encoder
WIV
Analysis Ki
Frame-level 9ynthesis NN
Timbre encoder
Giebal Timbow
Teatnable Liher
Eittedding
Anslysia KM
Timbre NB4
Figure 1:
Overview of proposed NANSY+ backbone architecture. All modules in the backbone network is trained in an end-to-end manner within single analysis and synthesis loop.
2.1 SELF-SUPERVISED LEARNING OF PITCH
We represent pitch with fundamental frequency Fo because explicitly controllable. In addition, we found that using sinsuoidal signal made by Fo as an input signal for synthesizer can greatly reduce glitches. To train Fo estimator in a self-supervised manner, we first adopt the auto-encoding approach of Engel et al. (2020b). Pitch encoder for takes Constant-Q Transform (CQT) as input feature and outputs probability distribution over 64 frequency bins where it spans from 50Hz to 1000Hz logarithmically (approximately 0.79 semitone per bin). The Fo is estimated by weighted averaging the probability distribution over the frequency bins. The pitch encoder also outputs two amplitude values, periodic amplitude An[n] and aperiodic amplitude Aap [n]. Fo [n] and Ap(m) linearly upsampled into a sample-level, Fo(t) and [1), and transformed into a sinusoidal waveform
x[|]] = Ap(t) sin [Ekn Folk) where N, denotes sampling rate. We also linearly upsample
2TT N.
[np[n] into Aap [t] and generate shaped noise y|4] Aap -n(4). where n[[] ~ U[-1, 1). Finally
the two siganls, z|t] and wit), added to form an input excitation signal [t] = xx(t] + yit] for a synthesizer. After the synthesizer reconstructs the input signal, the pitch encoder is trained with reconstruction loss.
The training solely depending on reconstruction, however, was unstable, and showed poor pitch estimation quality, which also reported in (Engel et al.) 2020b), and result they ended up utilizing synthetic audio datasets with pitch labels. While collecting synthetic audio datasets paired with pitch labels is not difficult for musical instruments, it is problematic for speech singing
2

---

voice because we cannot obtain such synthetic datasets. To address this issue, adopt another self-supervised pitch estimation method that utilizes relative pitch difference between two CQT inputs (Gfeller et al., 2020). First, we extract CQT X R RNXP, where N and F denote the size of time and frequency, respectively. One frequency bin of CQT feature amounts to 0.5 semitone. Next, we crop the frequency axis of X to obtain two CQT matrices with same size X X E
RR R2xx preape The frequency range of X consistently spans from Smin to Sman, whereas X is obtained by randomly shifting the frequency-axis index of X by d ~ U (dmin, dmax)). Then, the two cropped CQT features are passed through the pitch encoder and outputs two fundamental frequency sequences F(") and F Finally, relative pitch difference loss Spttch is computed as
Lpitch <(logg2 ( 1oga ( 0.5d], where h(-) denotes huber norm (Huber 1992). Note that by integrating two self-supervised pitch estimation methods into the end-to-end analysis-synthesis training loop, we achieve absolute pitch estimation without any synthetic datasets. and
follows:
For more detailed CQT configurations and neural architecture of pitch encoder, see Appendix A.1
2.2 DISENTANGLED LINGUISTIC REPRESENTATION
In order to ensure that the linguistic representation is disentangled from other analysis representa- tions, we combine information perturbation method proposed by Choi al. (2021a) and contrastive loss proposed by Qian et al. (2022). First, signal is transformed into two different signals using perturbation functions that does not significantly harm the linguistic information. Then, wav2vec features (Babu et al.) 2022) extracted from each perturbed signal are passed into linguistic informa- tion encoder fe.. Finally, the two linguistic representations, L!! contrastive loss as follows:
and L!?) are then used compute
exp cxp(d (a(En) _..T') 1k)
2
N
Lonntr Σ 1=1
log
(I)
Eve(nJUZ, exp (d(L!)) 1 /kk)
where l denotes a temperature parameter, d denotes cosine similarity, and In denotes the set of randomly selected frame indices for negative samples. The intuition of the contrastive loss is that the difference between (1) and (2) should be minimized, while the similarity within ( is min- imized that the consistent information within utterance (e.g., timbre information) can be min- imized. Note that we pass (1) to synthesizer so that feg is jointly trained with reconstruction loss within the whole end-to-end malysis-synthess loop, which is different from Qian et al. (2022) where they only focused on representation learning for discriminative tasks. As will show later in section 4.2 and |433 the learned linguistic representation is extremely helpful for fast training and data efficiency on synthesis tasks such TTS and SVS. For more details on information perturba- tion functions, neural architecture of linguistic information encoder, and hyperparameters for Leuutrr
see Appendix A.] and A.3
2.3 TIME- VARYING TIMBRE EMBEDDINGS
To synthesize waveform from disentangled pitch and linguistic representation, it is important to encode timbre information well enough fully reconstruct the input signal. Although the timbre information is usually encoded with a single vector, we assume that the capacity of the single vector not enough to capture the wide variety of timbral characteristics. In addition, we assume that timber is not static but rather varies over time, depending on time-varying contents such as pitch, amplitude, and linguistic information. Therefore, we breakdown timbre features into two global timbre embedding and timbre tokens using timbre encoder foy- Jor takes mel spectrogram as an input and produces global timbre embedding g, a single vector summarized with attentive statistical time-pon tioeeipooing (Okabe et al., 20188), which is expected to capture overall timbral characteristies within an utterance. The timbre tokens v, = I...., 1, are expected to capture diverse timbral character- istics into fixed number of tokens. The timbre tokens are extracted by adopting cross-attentin mechanism introduced by Jaegle et al. (2021), where trainable latent vectors are used as queries, and key and value are extracted from timbre encoder as illustrated in Appendix A.3 Fig.
After extracting timbre features, we adopt another cross-attention mechanism by Yin et al. (2022) to extract time-varying timbre embeddings that depends on other analysis content representations.
3

---

Specifically, we used the concatenation of [Fo. Ap. Asp- L, gi as query. Keys composed of I trainable vectors and the timbre tokens are used as values. Finally, the output of the cross-attention mechanism and g is interpolated using spherical linear interpolation to make tame-varying timbre emheddings. For more details of time-varying timbre embeddings, see Appendix A.3
2.4 WAVEFORM SYNTHESIS
The synthesis modules are composed of two synthesizers, frame-level and sample-level synthesizer. The frame-level synthesizer first takes linguistic feature and time-varying speaker embeddings to produce a frame-level condition for a sample-level synthesizer. After that, the frame-level condition are linearly upsampled into sample-level features for sample-level conditioning. The sample-level synthesizer then takes the excitation signal and sample-level features to reconstruct an input wave- form. For the sample-level synthesizer architecture, WaveGAN (Yamamoto et al., 2020). The whole analysis and synthesis modules are then trained with reconstruction loss. For the reconstruction loss, we used multi-scale spectrogram (MSS) loss (Wang et al.] 2019) and mel spectrogram loss (Kong et al.] 2020). Note that it was crucial to use linear frequency scale spectrogram for MSS loss rather than log-scale spectrogram for the stable training of pitch encoder, which shares similar observations to what (Turian & Henry (2020) has reported. We also used adversarial loss and feature matching loss for high quality synthesis. Multi- period discriminator (MPD) was used as discriminator architecture (Kong et al.) (2020). For more details of synthesizers, please refer to Appendix. Note that we downsample input waveforms into 16 kHz before passing into analysis modules and let the synthesizer produces the original 44.1 kHz waveforms. This enables 16 kHz to 44.1 kHz audio upsampling.
adopt the generator architecture of Parallel
3 EXPERIMENTS ON BACKBONE
We trained the backbone model on 10,571 hours (speech: 10,092 hours, singing: 479 hours) of proprictary 44.1 kHz audio recordings composed of 6,176 speakers and 624 singers. We trained the backbone model for IM iterations with Adam optimizer (Kingma & Ba 2014) with the learning rate of 10-4. The learning rate for MPD was 2 10 The batch size was set to 60 using 10 RTX 3090 GPUs
3.1 FUNDAMENTAL FREQUENCY DETECTION
To evaluate the robustness of the proposed pitch encoder in harsh conditions, we tested the performance of the pitch en- coder on noisy speech testset. We sampled 30 speech and noise recordings from the VCTK and DEMAND dataset (Veaux) (et al. 2017: Thiemann et al. 2013), respectively, and mixed them with 5 dB signal-to-noise ratio (SNR). For the baseline models, we chose 4 popular Fú estimators, that is, praat, rapt, pyin, and crepe (Boersma et al. 1993
Table
I: ABX results (%)
praat
rapt
pyin
crepe
89.8
88.8
84.6
71.1
Boersma & Van Heuven, 2001 Jadoul al. 2018 Talkin & Kleijn 1995 Mauch & Dixon 2014
Kim et al. 2018). We conducted ABX test, where 15 participants were asked which of the two samples (A and B) sounds more similar to the original sample (X). One of the two samples was gen- crated by reconstructing the original sample using the NANSY+ backbone, and another sample was reconstructed by simply replacing only the original Fo sequence into the ones that were extracted from one of the 4 baseline models. We conducted the experiments on Mechanical Turk (MTurk)). The results are shown in Table [ The results clearly show that Fu estimated by NANSY++ pitch encoder outperforms other pitch estimators significantly.
3.2 RECONSTRUCTION
In order to use the backbone synthesizer as a synthesis module for various applications, it is important to test the reconstruction performance. We randomly selected audio recordings for each of 25 unseen speakers and 25 unseen singers during training. 20 participants were involved for the evaluation on MTurk. As baseline model (BL;), we chose HiFi-GAN it is the most widely used peel-to-waeeforg converter (Kong et al. 2020). We trained HiFi-GAN using the same training set to NANSY++ For
Table 2: Reconstruction results.
Speech
Singing
GT
4.39
4.35
BL
3.26 4.37
4.00
Ours
4.36
4

---

fair comparison, doubled the upsampling rate of the penul- timate block of the HiFi-GAN generator, so that can produce 44. 14 waveform. The results are shown in Table 2 The results clearly show that NANSY++ synthesizer produces better waveforms for both speech and singing voice. Especially, significant improvement was observed on singing voice.
4 APPLICATIONS
Musical Score
Text
Reference Singer
NANSY-SVS
Sining Voie
Speech Synthess
NANSY-TTS
Reference Speaker
Systhesk
P/Ap Amplitude
+ Analyzer
Linguistic Feature
NANSY+
Wave
Synthesizer
Wave
Fg
Timbre Feature
Voice Attributes
NANSY-VOD
Voke Dasign
Figure 2: Overview of exemplar applications integrated into the NANSY+ backbone architecture. Each application can be substituted into a problem of estimating analysis features from the backbone.
We introduce exemplar applications that can be integrated with NANSY++. Various conditional generative models can be integrated by generating analysis features from conditions.
4.1 VOICE CONVERSION
We perform zero-shot voice conversion by simply replac- ing the original timbre features into the timbre features extracted from a single utterance of target speaker. To further match the style of the target speaker, we also transformed the F, statistics of the source speaker into the target speaker's. As a baseline model (BLy), we selected the state-of-the-art zero-shot voice conversion model, YourTTS (Casanova et al.] (022). Note that YourTTS requires paired (text, audio) dataset to train the model. Other than the NANSY++ trained with the entire dataset (Ours (entire)) explained in section 3 we trained additional model solely on VCTK dataset (Ours (vetk)) to match the training dataset setting of the baseline model. We tested each model using unseen speakers during training following EXP. of Casanova et al. (2022). Each speaker was converted into 4 different speakers, resulting in 44 conversion pairs. For each source-target con- version pair, we randomly selected utterances, resulting in 220 audio samples in total. We tested the naturalness (MOS) and speaker similarity (SSIM) following Wester al. (2016). The evaluation was done by 20 participants on MTurk. Note that we downsampled the 44.1 kHz NANSY+ outputs into 16 kHz for a fair comparison to YourTTS as only supports 16 kHz synthesis. The results are shown in Table their own utterances. The results clearly show that NANSY+ can change the voice with better nat- uralness and speaker similarity then the baseline model even though the model was trained without any text transcripts paired with audio recordings. addition, we could achieve a noticeable speaker similarity improvement when trained with more audio recordings, showing the data scalability of the training method of NANSY++.
Table 3: Voice conversion results.
MODEL
MOS
SSIM
TGT TGT
4.08
3.47
BLy
3.23
2.78
Ours (vctk)
3.72+m.06
2.99
Ours (entire)
3.79
3.16#0.10
TGT as TGT denotes the setting where the target speakers are compared with
4.2 TEXT-TO-SPEECH
In this section, deseribe NANSY-TTS, text to speech (TTS) application utilizing NANSY++ framework. NANSY-TTS is independently trained using NANSY++ analysis features. We show that this modular training approach is data efficient, while maintaining high-fideliy synthesis quality.
5

---

4.2.1 METHOD
Architecture NANSY-TTS consists of a phoneme encoder, style encoder, linguistic decoder, Fo decoder, amplitude decoder, and length regulator. All modules operate in a non-autoregressive man- ner. By utilizzig the timbre encoder of NANSY++ analyzer, there is no need to consider the timbre representation for NANSY-TTS, Therefore, the style encoder only handles the prosody (e.g., speak- ing pace, pitch, and loudness). The output style vector is used for duration predictor and all decoders. More details and the overview figure can be found at Appendix B.1
Training and inference To train the proposed TTS module, we optimize Mean Absolute Er- Tor(MAE) for linguistic features, Fu, Ap. and Aap extracted from NANSY++ analyzer. The aligner is trained independently to other modules in NANSY-TTS. The duration predictor is optimized by minimizing MAE for the duration of each phoneme using the alignment calculated by the monotonic alignment algorithm (MAS) (Kim et al.] 2020) with the aligner module. In the inference stage, the TTS module first generates linguistic feature, P/Ap amplitudes. and Fo. Then, NANSY++ synthe- sizer takes these generated features and timbre features to generate an waveform. More details can be found at Appendix B.2
4.2.2 EXPERIMENTS
Data efficiency To evaluate data effi- ciency, we used a single speaker (reader id: 8051) from the Hi-Fi multi-speaker english TTS dataset (Bakhturina et al. 2021). The dataset has 30 hours of speech audio sam- pled at kKZ. We made three subsets with data of 5, 10 and 30 minutes, and the larger subsets include small subsets. First, we mea- sured character error rate (CER) to evaluate the intelligibility of the speech according to the size of the dataset. The synthesized speech from the 100 test sentences were transcribed by the pretrained speech recognition model (Silero, 2021]. As baseline model (BLg), the official imple- mentation of Glow-TTS was used as an acoustic model. We used the same 44.1 kHz HiFi-GAN as baseline vocoder mentioned in section 3.2 The training configuration of NANSY-TTS (Ours) was set to be the same as that of the bascline model. As shown in Table 4 NANSY-TTS preserved relatively high MOS and low CER regardless of the size of the dataset, whereas evaluation results of the baseline model showed significantly dropped performance as the size of the dataset became smaller. In particular, NANSY-TTS trained with the 30 minutes subset outperforms the baseline model trained with the full dataset. This indicates the proposed modular training approach offers significant data efficiency for TTS with high synthesis quality.
Table 4: TTS data efficiency evaluation.
MODEL
MOS
CER (%)
GT
+.09
1.93
BLG / Ours (5)
1.37
33.31
42.50 3,29
BLG / Ours (10)
2.07 /3.39
18.59 3.05
BLG / Ours (30)
2.78-am1 / 33.44
5.34 / 2.20
BLG / Ours (full)
3.38-a.11 / 4.07
2.65 1.68
Zero-shot TTS The setting for this evaluation is the same section 4.1 but with three differences. First, the model generates speech samples using one ran- domly selected reference speech per speaker the style input. Second, we randomly selected 9 utterances for each of 11 unseen speakers, resulting in 99 audio samples in total. Finally, we trained MANSY-TTE on VCTK using two versions of NANSY++ as mentiond in section 4.1 respectively. As shown in Table NANSY- TTS outperforms the baseline. Interestingly, NANSY-TTS even achieved significantly better MOS than actual recordings. We conjecture that this is because some speakers in the VCTK datasets speak in rather unnatural prosody, while NANSY-TTS learns how to speak in natural prosody given the text transcription. We discuss the details in the Appendix B.3 further.
Table 5: Zero-shot TTS results.
MODEL
MOS
SSIM
TGT TGT
3.98
2.91
BLy
3.33
2.2 1000
Ours (vetk)
4.27
2.55
Ours (entire)
4.28
2.65.1009
4.3 SINGING VOICE SYNTHESIS
In this section, we describe a singing voice synthesis (SVS) application that generates N analysis features from musical score, namely NANSY-SVS. The two major challenges of SVS are (i) limited amount of labeled data and (ii) unpleasant glitches of synthesized voice. The first problem stems from the difficulty of dataset collection process. Labeling singing data often demands

---

more human labor and time compared to speech data. Another issue is that generating high-fidelity singing voice from mel spectrogram is well known to be difficult owing to a wide pitch range, long continuous pronunciation, and high sampling rate (Chen et al. 2020at Perrotin et al. 2021] Morrison et al., 2022). To tackle these problems, we introduce NANSY-SVS that builds upon the benefits of NANSY++ framework. NANSY-SVS addresses both concerns regarding the dataset size and high-fidelity synthesis by modeling the disentangled NANSY+ analysis features.
4.3.1 METHOD
Architecture NANSY-SVS uses a phoneme sequence, MIDI-pitch sequence, and global timbre embedding (from NANSY++ analyzer) as conditional inputs to generate linguistic features and Fo contours in autoregressive manner. The linguistic feature at the 1-th frame L, is inferred from Let, phoneme sequence, MIDI-pitch sequence, and singer-II) embedding. For the Fo contour, we predicted the difference (residual-Fo) between input MIDI-piteh and target Fo contour instead of modeling it directly. The residual-Fo range ending in - -1200 cents to 1200 cents is quantized in 100 cents increments, yielding 241 possible values. The 241-dimensional residual-F'. RF,, was inferred from RFct, MIDI-pitoh sequence, phoneme sequence, and singer-ID embedding. The amplitude predictor module infers P/Ap amplitudes from the generated linguistic features and Fo contour. More details of NANSY-SVS can be found in Appendix C.I
Training and inference To train NANSY-SVS, we optimize MAE for linguistic features, P/Ap amplitude, and cross-entropy for residual-Fo classification. At inference time, linguistic features and residual-Fo are first generated in autoregressive way, and then P/Ap amplitudes are inferred from the generated results. Finally, the generated features passed through the NANSY+ synthesizer along with the timbre features extracted from the reference audio to produce a final waveform. 4.3.2 EXPERIMENTS
Experimental setup We trained the model using OpenCPOP dataset (Wang al. 2022 a public singing dataset of 100 songs from single female singer with aligned phoneme and MIDI-pitch sequence. We followed the official train/test split of Wang al. (2022), i.e. 95 songs and 5 songs for training and evaluation, respectively. To verify that our proposed model can be trained to syn- thesize high-quality singing voice using a limited amount of training data, we constructed a subset by extracting 80, 160, and 320 segments from the original dataset consisting of 3550 segments in total. This corresponds to 10%, 5%, and 2.5% of the entire dataset, and we trained 4 models in total using each subset. Note that each subset was constructed to include all phonemes and MIDI-pitch at least once. As a baseline model (BLD), we chose Diffsinger (Liu et al., 2022). which showed state-of-the -art performance with diffusion-basel model structure. We trained the baseline model with the same setting as NANSY-SVS using the official implementation for both acoustic model and vocoder.
Evaluation We conducted a listening test eval- uate the quality of generated singing voice. We ran- domly selected 12 segments from each of the test songs, and generated 60 singing voice segments in total for each model. The singing-ID embedding was obtained from a randomly sampled 30-second segment in the training set. 15 participants were asked to assess the naturalness of each audio a 5-point scale. Table 6 shows that the perceptual quality of the NANSY-SVS outperforms the base- line in all training conditions. As the size of the training data decreased, the MOS of Diffsinger decreased by up to 1.16 while the MOS of NANSY-SVS decreased by only 0.06. From this, we confirm that the proposed modularized training method with NANSY+ backbone is efficient in data-limited training condition. Other experiments on SVS can be found Appendix C.2
Table 6: SVS data efficiency evaluation.
MODEL
MOS
BLD Ours (60)
2.67.1088 1
BLD Ouus (160)
3.05 / 3.86600005
BLD Ours (320)
3.49 3.85
BLD / Ours (full)
3.83 +aus / 3.87
GT RTcRe
4.026+0as6 / 4.033 +ouss
4.4
VOICE DESIGNING
To further explore the usability of disentangled NANSY+ features, we tackle the challenging prob- lems of manipulating voice such as creating new voice identities and editing voice attributes. Pre- vious works have investigated extrapolating of speaker embedding space using Gaussian mixture models (GMMs) (Bilinski al. 2022) learning a speaker embedding prior within a variational
7

---

honder sample.
Edtimated
62-
200
130
46
30
20
-.5
49
Age shift
Gender shif:
Figure 3: Age shift evaluation.
Figure Gender shift evaluation.
autoencoder (VAE) framework (Stanton et al. 2022) to synthesize new speakers. this section, we develop a voice designing system, namely NANSY-VOD, that can both change voice attributes and generate novel voice identities. NANSY-VOI VOD also enables fine-grained control over voice manipu- lation in an interpretable way by leveraging speaker's gender and age conditioning variables.
4.4.1
METHOD
Model NANSY-VOD consists of three normalizing flow networks that learn the distribution of Fo statisties (e.g.. median and variance), global timbre embedding, and timbre tokens, respectively. The first network estimates the conditional distribution of the Fo statistics given gender and age, and is trained according the maximum likelihood. The second network is optimized maximize the likelihood of the global timbre emhedding conditioned on the Fb statistics, gender, and age. Similarly, the third network models the distribution of the timbre tokens given the global timbre embedding, the Fo statistics, gender, and age. Since the timbre tokens consist of multiple column vectors lying common feature space, we choose to use each vector as individual input to the network. To distinguish these vectors, a token index is used as additional conditioning term. By sharing parameters, the third network trained more efficiently compared to the naive implementa- tion that models the entire timbre tokens at once. More details can be found in Appendix D.I
Continuous gender We continuous real values instead of binary values to represent gender. The intuition is that (i) voices of pre-pubertal children may not be gender distinct, and (ii) some peo- ple's voices sound neutral. By allowing relaxed labels for gender, we expect that the gradual control of the gender attribute of voice can be implemented at inference time. To get continuous gender labels, another network called SPNet is employed and more details can be found at Appendix D.4
Training and inference Input data is obtained from NANSY ++ analysis modules and condition- ing data (i.e. gender and age) is labeled by SPNet for training. The negative log likelihoods of the three networks are summed to compute the total loss and NANSY-VOD is optimized in the same way typical normalizing flow. At inference time, NANSY-VOD supports attribute control and new voice generation. To edit voice attributes, first, the inputs are transformed flow latent variables conditioned on the source speaker's gender and age. The edited NANSY++ features are then retrieved by converting the latent variables back conditioned on different age and gender. To create a new voice identity, latent variables are sampled from the normal distribution and transformed via the inverse pass of MANSY-VOD with target age and gender.
functionalities: voice
4.4.2 EXPERIMENTS
Experimental setup To evaluate the proposed method, we used the NANSY+ backbone dataset and three Al-Hub conversation datasets (https://aihub.or.k
kids, adults, and elderly people. The AI-Hub datasets consist of approximately 6821 hours of Korean recordings with gender and age labels. There are total of 6001 speakers whose ages range from 3 to 91. We randomly selected 4800 and 600 speakers, and constructed training and validation sets respectively by merging their utterances and speech data of the NANSY++ backbone dataset. From the remaining 601 speakers, we prepared a set of 320 unseen speakers by sampling 40 male and female speakers from each age group of (0, 10), (10, 30), (30, 50], and (50, 70]. We trained NANSY-VOD for about 250K iterations with the Adam W optimizer ( (Loschhloo & Hutter (207)) of learning rate x 10-4, A batch size was to 256 for each of four RTX 3090 GPUs. Other experimental settings such as model architecture and hyperparameters are attached at Appendix D.I
Voice attribute control NANSY-VOD controls voice attributes by using different conditioning
inputs for the forward and inverse processes of normalizing flows. First, edited the age attributes
8

---

of the 320 unseen speakers by adding subtracting different values to their original ages. The synthesizer reconstructed audio using the linguistic feature of a source speaker and the other edited features. Then, the ages of the converted voices were estimated by SPNet. We report the estimated age distributions of the original voices and the edited voices in Fig. 3 A shift in the age distribution was observed as the target age shifts, supporting the age controllability of NANSY-VOD.
We also edited the gender attribute and investigated the gender of the converted voices using SPNet. Since NANSY-VOD uses a continuous gender. SPNet was used to label the gender of a source speaker. Fig. A shows the gender distributions of the original and converted voices. We observed the gender distribution varies as we shift the gender values accordingly. This shows that NANSY- VOD can transform a voice into more masculine or feminine voice by controlling the gender attribute.
Voice identity generation We evaluated the perfor- mance of voice identity generation in terms of speaker diversity. To measure how diverse speakers NANSY. VOD can generate, we utilized the speaker distance metric proposed by Stanton et al. (2022), which refor- mulated as median min d( Vy), where d(.,.) de-
Table 7: Speaker diversity. n, denotes the number of speakers in comparison set.
n,,320
n,=240
n,,160
n==1 Ours
0.51
0.46
0.39
0.17
0.46
notes the cosine distance and VÚ represents the speaker feature of the i-th utterance extracted from ECAPA-TINN (Desplanques et al. 2020). We synthe- sized 320 speakers from NANSY-VOD and synthesized audio using the linguistic features of the utterances in the test set. For comparison, we constructed four sets of 320 utterances consisting of different numbers of speakers (320, 240, 160, and 1). Table 7 shows the speaker diversities of the set generated by NANSY-VOD and the other sets. The set consisting of 320 speakers records the highest diversity and we use this score as estimate of the speaker diversity of the real speakers. As the number of speakers in the set decreases, the speaker diversity gets reduced. The speaker diversity of NANSY-VOD is close that of the 240 speakers set. This implies that NANSY-VOD can generate various speakers having 75% diversity compared to the real speaker distribution.
RELATED WORKS
Self-supervised representation training methods for speech has recently been widely studied in the goal of training rich representations mostly for downstream tasks such automatic speech recog- nition (Bacvski et al. 2020; Babu et al. 2022, Hsu et al., 202T Huang et al., 2022). There has also been studies that utilized the self-supervised speech representation on synthesis task such as voice conversion (Lin et al. 2021 hao Lin et al. (021). For TTS applications, Siuzdak et al. (2022)
proposed employ wav2vec 2.0 feature as a mid-level representation of TTS model. However, they used the representation from wav2vec 2.0 network finetuned on English text transcriptions, hence might not be sufficient for training on languages without labeled datasets. Kim et al. (2022) has proposed to apply transfer learning framework using pseudo phoneme sequence extracted from wav2vec 2.0 feature and showed that the model can be trained using a small amount of dataset.
Previous works have shown that state-of ofte-oofthhe-rt vocoder such as HiFi-GAN produces glitches, espe- cially on long notes (Morrison et al., 2022; Chen et al. 2020a). Morrison et al. (2022) showed that an autoregressive neural architecture can reduce this issue. Another line of vocoders are the ones that take sinusoidal signal as a network input (Wang et al.] 2019 Hono et al. (02T). These types of vocoders do not exhibit such glitches on long notes even without the autoregressive architecture.
6 CONCLUSION
In this work, we proposed a unified voice synthesis framework NANSY++ that can analyze signal into disentangled representations and synthesize high-quality waveform. Because the training only requires audio recordings, we can achieve data scalability. We have also proposed to integrate various voice synthesis applications in modularized way. By integrating the applications into the centralized backbone network, we found that we can train models for each application using only a small amount of dataset. Although each module is not trained in an end-to-end manner, the proposed modularized training strategy does not suffer from the common performance drop phenomenon induced by training/inference mismatch. We believe that the proposed framework provides useful representations for synthesis tasks. Therefore, we hope that the proposed framework will help to move away from the current predominant training strategy (text-to-mme acoustic model & mel-to-way vocoder) and expand the scope of research.

---

7 REPRODUCIBILITY STATEMENT
Details of neural architectures for every models used this paper are described in Appendix. We have also included every hyperparameters used for training in Appendix.
ETHICS STATEMENT
There is a possibility of the proposed method being used with harmful intentions by anonymous users. Because the proposed method can clone arbitrary voice in zero-shot manner, we believe the technology should be released only to the identified and authorized users.
As the voice synthesis technologies advance rapidly, attempts have been made to counteract the concerns surrounding the harmful intents of the voice synthesis technology. Recently, anti-spoofing challenges have drawn attentions from many fields because of its importance, and showed promising results anti-spoofing for speech (Yamagishi et al. 2021). We hope engage and develop voice verification system to prevent abuse of fake voice.
more domain experts to
Despite the concerns negative consequences of the technology. there is also bright side of such technologies when used properly. One such example is when used as an interactive tool between humans and machines. Because the proposed methods enable controls on voice, believe it can foster human creativity in various aspects.
REFERENCES
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kri- tika Singh, Patrick Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. In Proc. Interspeech 2022, pp. 2278-2282, 2022. doi: 10.2445/lmeersecchh.222211111
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33: 2449-12466060 2020.
Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. Hi-fi multi english Its dataset. arXiv preprini arXiv:2104.01497 2021.
Piotr Bilinski, Thomas Merritt, Abdelhamid Ezzerg, Kamil Pokora, Sebastian Cygert, Kayoko Yanagisawa, Roberto Barra-Chicote, and Daniel Korzekwa. Creating New Voices using Nor- malizing Flows. In Proc. Interspeech 2022, pp. 2958-2962, 2022. 2022-10195.
doi: 10.21437/nntrspeeech.
Mikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with ad- versarial networks. In International Conference on Learning Representations, 2020. URL
Settps://openreviee
net/ Jett//oruunio--lggg033
Paul Boersma and Vincent Van Heuven. Speak and unspeak with praat. Glot International, 5: 341-347, 2001.
Paul Boersma al. Accurate short-term analysis of the fundamental frequency and the harmonics- to-noise ratio of sampled sound. In Proceedings of the institute of phonetic sciences, volume 17, pp. 10. Citeseer, 1993.
Edresson Casanova, Julian Weber, Christopher D Shulby, Amaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pp. 2709-2720. PMLR, 2022.
Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie-Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint AXXi:20099001770 2020a.
Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Tie- Yan Liu, et al. Adaspeech: Adaptive text speech for custom voice. In International Conference Learning Representations, 2020b.
10

---

Hyeong- -Song Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. Neural analysis and synthesis: Reconstructing speech from self-supervised representations. Advances in Neural Information Processing Systems, 34: 14:11221-16265 2021a.
Hyeong-Seok Choi, Sungjin Park, Jie Hwan Lee, Hoon Heo, Dongsuk Jeon, and Kyogu Lee. Real- time denoising and dereverberation wtih tiny recurrent u-net. In ICASSP 2021-2021 IEEE In- ternational Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5789-5792 IEEE, 2021b.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941. PMLR, 2017.
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck, ECAPA-TINN: Emphasized Chan- nel Attention, propagation and aggregation in TDNN based speaker verification. In Interspeech 2020, pp. 3830-3834, 2020.
Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial sppecc.. In International Conference on Learning Representations, 2021. URL
https:://gpenrevilww net/ of
Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable
digital signal processing. In International Conference on Learning Representations, 2020a. URL
httpps///penrevieew.rr net/ htt///orrmiiddd1-11
Jesse Engel, Rigel Swavely, Lamtharn Hanoi Hantrakul, Adam Roberts, and Curtis Hawthorne. Self-supervised pitch detection by inverse audio synthesis. In ICML 2020 Workshop on Self- supervision in Audio and Speech, 2020b. URL httph:://////.... Tettps::/openreviie
net/ forum?ide
R1VTYWhsky7
Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mibajlo Ve- limirović. Spice: Self-supervised pitch estimation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1 1e:h1118-1111 11-1188-18811 2020.
Jheng hao Lin, Yist Y. Lin, Chung-Ming Chien, and Hung yi Lee. S2VC: A Framework for Any- to-Any Voice Conversion with Supervipe Pretrained Representations. In Proc. Interspeech 2021, pp. 836-840, 2021.
doi: 0022443/lnterspeechh2222 1356.
Yukiya Hono, Shinji Takaki, Kei Hashimoto, Keiichiro Oura, Yoshihiko Nankaku, and Keiichi Tokuda. Periodnet: non-autoregressive waveform generation model with structure separating periodic and aperiodic components. In ICASSP 2021-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6049-6053. IEEE, 2021.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions on Audio. Speech, and Language Processing, 29:3451-3460, 2021.
Wenyong Huang, Zhenhe Zhang, Yu Ting Yeung, Xin Jiang, and Qun Liu. SPIRAL: Self-supervised perturbation-invariant representation learning for speech pre-training. In International Confer- ence on Learning Representations, 2022. URL
net/ fortt/formmmid
TBpg4PnXhYHl
Peter J Huber. Robust estimation of location parameter. In Breakthroughs in statistics, pp. 492- 518. Springer, 1992.
Yannick Jadoul, Bill Thompson, and Bart De Boer. Introducing parselmouth: A python interface to praat. Journal of Phonetics, 71:1-15, 2018.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Car- reira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4651 4664. PMLR, 18 24 Jul 2021. URL https:://rocees
www.ppresssse vt/ jaeg e21 22........ htmll

---

Myeonghun Jeong. Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-ho-speech. arXiv preprint arXiv:2104.01409, 2021.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Sys- tems, 33:8067-8077, 2020.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference Machine Learning, pp- 5530-5540. PMLR, 2021.
Jong Wook Kim. Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional repre- sentation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165. IEFF, 2018.
Minchan Kim, Myeonghun Jeong, Byoung Jin Choi, Sunghwan Ahn, Joun Yeop Lee, and Nam Soo Kim. Transfer Learning Framework for Low-Resoures Text-to- Text-to-Speech beled Speech Corpus. In Proc. Interspeech 2022, pp. 788-792, 2022. doi: 20.21437/nterrpeechh. 2022-225.
Speech using Large-Scale Unla-
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint /4/2.644/2.69000 2014.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033. 2020.
Juheon Lee, Hyeong-Seok Choi, Chang-Bin Jeon, Junghyun Koo, and Kyogu Lee. Adversari- ally trained end-to-end korean singing voice synthesis system. In Germot Kubin and Zdravko Kacic (eds.), Interspeech 2019, 20th Annual Conference of the International Speech Commu- nication Association, Graz, Austria, 15-19 September 2019, pp. 2588 2592. ISCA, 2019. doi: 21437/ URL htttts::s//oiiiii org 2t0p:///0002443////eeeeeec.. Interspeech.
00.21437/lneerpecch200.99 2019-1722
Yist Y Lin, Chung-Minn Ming Chien, Jheng-Hao Lin, Hung-yi Lee, and Lin-shan Lee. Fragmentve: Any- to-any voice conversion by end-to-end extracting and fusing fine- ane-grained voice fragments with attention. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5939-5943. IEEE, 2021.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice syn- thesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11020-11028, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Matthias Mauch and Simon Dixon. pyin: A fundamental frequency estimator using probabilistic threshold distributions. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pp. 659-663. IEEE, 2014.
Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. Meta-stylespeech: Multi-speaker adaptive text-to-speech generation. In International Conference on Machine Learning, pp. 7748- 7759. PMLR, 2021.
Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. Chunked autoregressive GAN for conditional waveform synthesis. In International Conference Learning Representations, 2022. Utthttps:: htattps://openreviiew net forum?
id=v3aelsY_wVV
Koji Okabe, Takafumi Koshinaka, and Koichi Shinoda. Attentive statistics pooling for deep speaker embedding. In Proc. Interspeech 2018, pp. 2252-2256, 2018. doi: 10.21437/nttrspeccch 2018-993.
12

---

Olivier Perrotin, Hussein Amouri, Gérard Bailly, and Thomas Hueber. Evaluating the extrapolation capabilities of neural vocoders to extreme pitch values. In Interspeech 2021, pp. 11-15. ISCA,
2021.
Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa- Johnson, and Shiyu Chang. Contentvec: An improved self-supervised speech representation by disentangling speakers. In International Conference on Machine Learning, pp. 18003-18017. PMLR, 2022.
Silero.
Silero models: pre-trained enterprise-grade stt / tts models and benchmarks. htgithuub.com//snkersrstte
https:
mooo0mooolel. 2021.
Hubert Siuzdak, Piotr Dura, Pol van Rijn, and Nori Jacoby. Wavthruvec: Latent speech represen-
tation intermediate features for neural speech synthesis. arXiv preprint arXiv:2203.16930, 2022.
Daisy Stanton, Matt Shannon, Soroosh Mariooryad, RJ Skenyy- Eric Battenberg, Tom Bagby, and David Kao. Speaker generation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7897-7901. IEEE, 2022.
David Talkin and Bastiaan Kleijn. A robust algorithm for pitch tracking (rapt). Speech coding and synthesis, 495:518, 1995.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. Demand: a collection of multi-channel recordings of acoustic noise in diverse environments. In Proc. Meetings Acoust, pp. 1-6, 2013.
Joseph Turian and Max Henry, I'm sorry for your loss: Spectrally- based audio distances are bad at 2020.
pitch. arXiv preprint arXiv:2012.088
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Cstr vetk corpus: English multi- speaker corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2017.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for sta- tistical parametric speech synthesis. IEEE/ACM Transactions Audio, Speech, and Language Processing, 28:402-415, 2019.
Yu Wang. Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. arXiv preprint arXiv:2201.0744 2022.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint 1703 2017.
J Weiss, RJ Skerry-Rya Eric Battenberg, Soroosh Mariooryad, and Diederik P Kingma. Wave- tacotron: Spectrogram-free end-to-end text-to-speech synthesis. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5679-5683 IEEE, 2021.
Mirjam Wester, Zhizheng Wu, and Junichi Yamagishi. Analysis of the voice conversion challenge 2016 evaluation results. In Interspeech, pp. 1637-1641, 2016.
Junichi Yamagishi, Xin Wang, Massimiliano Todisco, Md Sahidullah, Jose Patino, Andreas Nautsch, Xuechen Liu, Kong Aik Lee, Tomi Kinnunen, Nicholas Evans, and Héctor Delgado. Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection. In Proc. 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge, pp. 47-54, 2021. doi: 10.2143/ASVSPPOFF.D0I1
13

---

Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 61 6199-6203. IEEE, 2020.
Dacheng Yin, Xuanchi Ren, Chong Luo, Yuwang Wang, Zhiwei Xiong, and Wenjun Zeng. Re- triever: Learning content-style representation as token-level bipartite graph. In International Conference on Learning Representations, 2022. URLhttps: openreview net / forum? Id=AX00ygMuT66]
14

---

A BACKBONE
A.I TRAINING DETAILS
Linguistic feature In the training stage of the backbone architecture, important perturb the information before extracting the linguistic feature so that it does not contain any information related timbre. We adopted the four perturbation functions i.e. formant shift, pitch randomiza- tion, parametric equalizer- used in Choi et al. (2021a) along with two more perturabtion functions,
breathiness perturabtion, and additive noise. The noise was randomly selected from the DEMAND dataset (Theemann et al. 2013).
We decide indice set Z, for negative samples by generating random mask that does not contain nearby 10 frames of the linguistic feature that the negative frames not sampled from the ones that contain the same linguistic characteristics to the positive frame. We set the temperature to 0.1
and the coefficient for Leontr was linearly scheduled from 10-5 to 10, following (Qian al. 2022
Sele-supe Fo estimation The configurations for CQT E RXXF and cropped CQT ( (2) IR N as follows. The number of bins per octave is 24. The frequency range spans from 32.7 Hz to 8000 Hz. The size of frequency-axis of CQT feature before crop F is 191. The frequency size of the cropped CQT features Esecpp 160. The minimum and maximum bound for sampling distribution d ~ 21 (dming dmax) for index shift was set to -12 and +12, which amounts -6 and semitone.
A.2 QUALITATIVE RESULTS ON PITCH ESTIMATION
RAPT FO ast mation
NANSY+ essimation
16384
-30
10.004
1:0 49
13 65
130 68
2045
1024
1024
317
-10 30
532
-10
238
-20
64
08
-0 3B
Tiere
RAFT estimation
NANSY estimation
150 68
8152
0302
-20 05
4986
4740
2018
2048
+30 05
J2
10 16
235
120
320
d8
-30
0.5
3.5
2.4
4.5
0.5
2.2
2.0
2.1
3.5
4.8
Figure 5: The qualitative results Fo estimation. Left column shows the Fo estimation results from rapt algorithm. Right column shows the Fo estimation results from NANSY++ pitch encoder. First row shows the Fo estimation results from clean signal. Second row shows the F/ estimation results from noisy signal. The results clearly demonstrates that the pitch encoder is indeed estimating Fo. robustly for noisy signal.
15

---

In order to show that the output from the pitch encoder is really estimating fundamental frequency, we show the results of the estimated fundamental frequency Figure We can see that for clean signals, rapt algorithm and NANSY++ pitch encoder estimating the same Fo trajectory, showing that the pitch encoder trained in a self-supervised manner is actually estimating Fo. Additionally, noisy signal to demonstrate the robustness of the proposed
have also tested the algorithms pitch encoder on noise. The second row shows that NANSY+ pitch encoder can estimate Fo even the harsh environment, whereas rapt algorithm completely fails.
A.3 NEURAL ARCHITECTURES
Pitch encoder For the architectural design of pitch encoder, we used ID-CNN blocks operating in frequency-axis combined with bi-directional gated recurrent unit similar to Choi et al. (2021b). The architecture is illustrated in Fig. 6 For the last activation function of P/Ap amplitude heads,
used exponential sigmoid used in Engel et al. (220a))
Output O.N]
prob. [B. 64,N]
aamp [B. L.M]
[B. 1.
Softmax
Exp. Sigmold
Exp. Sigmold
Pool (x 0.5)
Ap
Head
Head
Head
Linear (C, O)
Convid (k, 1,
ReLU
GELU
Linear (H, 2C)
BatchNorm
Hidden [B.0,M N
GRU
Convid (k, 1, 1)
Reshape 128, F//4, N] [B, N, 32
GELU
ResBlock (3, 128)
BatchNorm
ResBlock (3, 128)
Input [,N
Convid (7,
Input [B, Eroper N]
ResBlock (k, O)
Pitch encoder (b)
(a)
Figure 6: Pitch encoder architecture. k, d, s in Convld ld (k, d, s) denotes kernel size, dilation, and stride. k, O in ResBlock (k, O) denotes kernel size and output channel size. C, O in Linear (C, O) denote input and output channel size.
Linguistic information encoder For the architectural design of linguistic information encoder, we used Convolutional Gated Linear Unit (ConvGLUU) blocks (Dauphin et al. 2017). The architecture is illustrated in Fig. 7
The architectural details of timbre encoder and time-varying timbre encoder are shown in Fig. 8 We borrowed the neural architecture for timbre encoder from ECAPA-TDNN (Desplanques et al.) 2020). The timbre tokens are extracted via cross-attention mechanism, where 50 trainable latent vectors are used a query, and features from multilayer fea- ture aggregation (MFA) block of ECAPA-TDNN is transformed into key and value. We also extract global timbre embedding via attentive statistical pooling (ASP). Time-varying timbre en- coder then utilizes another cross-attention mechanism by taking content queries, concatenation of [Fo, Ap: Aap. L, g). trainable 50 latent keys, and timbre tokens values. The output of the cross- attention and global timbre embedding is then interpolated on a spherical surface to produce the final time-varying timbre embeddings.
Time-varying Speaker Embeddings
16

---

Output N,
Output [B, C.N]
Linguistic F Feature [B, 128,
1 Normalization
Dropout
GLU
Convtd (1, 1,
2C
ConvGLU (kr1, d=1, s=1)
Lea
Convid d,
x2
ConvGLU [Km 6m1, 5-1)
Convid (1,
Dropout
x
(128, 128)
Input [B,C,N]
Input [B, N]
PreConv (1024, 128)
Wav2vec-15th [B. 1024,N]
PreConv (c', (a)
ConvGLU (b)
Linguistic Information Encoder
(c)
Figure 7: Linguistic information encoder architecture. C' and in PreConv denote input channel and output channel size. k, d, in Conv 1d (k, d, s) denotes kernel size, dilation, and stride.
Timbre Takens 58)
Timbre Tokens Global Timbre Embeddding
Time-varying Timbre Embeddlings [u. 192.N]
[R. 128, 50]
192.1]
Slerp
Tile
bg Norm.
Mult-Head Attention
Linear
1g Norm.
Global Timbre Embedding [R. 192 N]
TTB
ASP
Linear (128, 192)
MFA
Mutt-Head
Multi-Hede Attention
Latent Query 58.512.50
MFA Feat.
Content Queries
Latent Keys Timbre Takens [28,550
[8.3072NN]
Mel Spectrogram N)
128 50)
[B,323
Timber Token Block (TTB)
Timber Encoder (b)
Time-varying Timbre Encoder
(a)
(c)
Figure& 8: The architecture of timbre encoder and time-varying timbre encoder. MFA denotes Mul-
tilayer Feature Aggregation block of ECAPA-TDNN, ASP denotes Attentive Statistical Pooling.
Synthesizer
The architectural details of frame-level and sample-level synthesizers are shown in Fig. [A.]] The frame-level synthesizer takes the linguistic feature and time-varying speaker em- beddings and outputs frame-level condition. The frame-level condition is then passed through the sample-level synthesizer along with Fo. periodic amplitude (P amp), aperiodic amplitude (Ap amp) to produce waveform. We borrowed the neural architecture of parallel wavegan (PWGAN) for the sample-level synthesizer (Yamamoto et al. 2020).
B TEXT-TO-SPEECH
B.1 DETAILED ARCHITECTURE
Encoder The architectural details of encoders shown in Fig. I1 Similar to Min et al. (2021) NANSY-TTS uses the style encoder and conditional layer normalization (cLN) to train the various prosody. However, the hidden feature extracted from the third layer of wav2vec network
17

---

Output N.
Output [B,
Frame-loval Condition (R.C.N)
Weveform [B. [B,1,T]
Time-varying
Timbre
Convid (1, 1)
Dropout
Embeddings
Frame-level
cLN (optional)
ConvGLU [el sel,
Condition [B.C.]]
LeakyRalU
GLU 22
ConvaLU
Simusoid Noise
Canvid (1, 1,
Convid [K,
ConvGLU Bes3) xrt d
Input
Dropout
Ap
PreConv
[B. 1,
[0.1.N]
[B.1.N]
Time-varying Timbre Embeddingg
Lingustic JB, C.N]
Input [B.
PreConv (a)
ConvGLU (b)
Frame-level Synthesizer
Sample-level Synthesizer
(d)
Figure 9: The architecture of frame-level synthesizer and sample-level synthesizer. cLN denotes conditional Layer Normalization.
(Babu et al. 2022 instead of mel spectrogram as the style input. The output vector encoded by the style encoder used by all cLNs in NANSY-TTS. The phoneme encoder is composed of lookup embedding table, 3 CenvReLUNorm blocks, 3 Transformer Vaswani al. 2017) blocks, and a final linear layer with LeakyReLU.
Decoder The architectural details of decoders are shown in Fig. 12 First, the phoneme feature sequences are upsampled using the duration predicted by the duration predictor extracted by the aligner. The upsampled sequences are then fed to all decoders. The amplitude decoder is composed of ConvRel .NNorm blocks with cLNs and a final linear layer. To predict Fo more elaborately, the Fo decoder also use the hidden features from the amplitude decoder an additional input. The Fo decoder consists of CeLuRem blocks with cLNs, GRU and final linear layer.
Aligner We train an aligner independently of the TTS model. It consists of an encoder and decoder. The encoder is composed of 5 Re
and 1) blocks shown in Fig.
II The decoder is the same as in Kim al. (2020) except for three differences. First, has only 2 blocks instead of 12 blocks of the original version. Secondly, it uses the linguistic feature as the output instead of mel spectrogram. Finally, the aligner including the decoder is only used during training, and the value calculated by MAS is used as the phoneme duration.
Phaneme Sequenca
Phoname Encoder
Linguistic Decoder
Linguistic Feature
Amplitude Decoder
P/Ap Amplitude
NANSY
Synthesizer
Wave
Wava
Style Encoder
Fg Decoder
Contour
Timbre Encoder
Timbre Feetures
Figure 10: Overview of NANSY-TTS.
B.2 TRAINING DETAILS
All models were trained for 200K iterations using Adam optimizer (Kingma & Ba 2014) with the learning rate of 10-4. In the training stage, we use randomly sliced ground truth waveform as the style reference input. This helps to ensure robust performance
when nonparallel reference, which has a different content from the input text, is used as the style input in the inference stage. To train the prosody components efficiently, apply the min-max normalization for A Aap and Fo.
18

---

Output NB.C.N Nuat/S
Stylc Embodding 12
Output [M.
Phoneme Festure [B. N20l
LeakyReLU
CL.N
Dropout
Lineer
Linear
Linear
LayerNorm
Transformer Block
Lea
cLN
x3
Convid (K. d.
ConvRell
Dropout
ConvReLUNorm (edlact
x3
Transformer Layer
Embedding
Input Maal
Wev2Vec-3rd [B, N]
Style Embedding
Input [,
Style Embedding
Phonome
[R, N.perl
ConvRaLUNorm
Style Encoder (b)
Transformer Block
Phoneme Encoder
(a)
(c)
(d)
Figure 11: NANSY-TTS encoder architecture. k, d, and s of the Conv 1d layer denotes kernel size, dilation, and stride, respectively.
Duration [R. N.atl
Lingurstie
amp / Ap [0.2.M]
128.N]
[R.
Convicuant sill
Lineer
Convidie (-l)
Linear
GRU
UNorm
cLN
DonvielLLNom (i-h)
Transformer
ConvRic
2-2 m-
CamvReLLNNamm (b+6, (-1)
Phanema Feature [It 12% Nucil Duration Predictor (a)
Style Embedding
Upsampied Feature
Upsampled Phoname Feature
Styla Unsampled Phonema Feature
[0.128
N] Amplitude Decoder (c)
Embediinng
[B, [8,120.
Linguistic Decader
FO Decoder
(b)
(d)
Figure 12: NANSY-TTS decoder architecture.
B.3 DETAILED EVALUATION RESULT
Training efficiency To verify the efficiency of train- ing time, measured character rate (CER (%) each training step. The experimental setting is the same the setting for the full single-speaker dataset in sec- tion 4.2.2 As shown in Fig. 13 NANSY-TTS achieved low CER even though it was tramed for only few thou- sand steps. In comparison, the bascline model (Glow- TTS) showed relatively slower training convergence. In particular, NANSY-TTS trained for only 2k steps out- performed Glow-TTS trained for 200k steps. addi- tion, NANSY-TTS trained for 10k steps achieved lower CER than actual recordings. We speculate that this because the NANSY++ analysis features are easier to model compared to the mel spectrogram, which is an entangled representation of amplitudes, pitch, linguis- tie information, and timbre.
<10.24
6.68
3.40
2.34
2,00
1.7
1.68
Taining stepod
Figure 13: Fast training convergence.
Zero-shot TTS results for each speaker As shown in Table 5 NANSY-TTS achieved better
MOS than actual recordings. This means that the raters felt that the speech generated by NANSY-
19

---

TTS natural than the actual recordings. Actual recordings often have diverse styles (e.g., speaking speed, pauses, intonation, etc.), which made the listener feel unnatural. In particular, noticeable in certain speakers of VCTK dataset, as shown in Table g The audio samples provided in the demo page: finyurl.com/Stnsyyuu
Table 8: Zero-shot TTS results for each speaker. MOS(left) and SSIM(right)
Model Accent
GT
YourTTS
NANSY-TTS
NANSY-TTS (entire)
Speaker
(vetk)
p225
English
3.93/2.67
3.27 / 2.56
4.29/2.93
4.40/3.04
p234
Scottish
3.89/2.47
3.91/2.64
4.31/2.73
4.31/3.27
p238
NorthernIrish
4.22/3.11
3.58
4.31/2.24
4.36
p245
Irish
4.29/2.44
3.11/2.07
4.36/2.42
4.11/2.42
p248
Indian
3.60/3.13
2.93/
4.33/ 1.80
4.36
p261
NorthernIrish
4.09/2.98
2.89/1.9
4.20/2.84
4.33/3.29
p294
American
4.16/3.00
3.84
4.16/3.20
3
p302
Canadian
3.80/2.69
/.000
4.44/2242
4.36
p326
Australian
3,91/3.22
1.96
4.38 /316
4.20 /3.27
p335
NewZealand
3.93/3.13
3.00
4.22 2.38
4.33 /.113
p347
SouthAfrican
3.96/3.18
3.64 2.09
4.02 1.93
4.38 2.16
C SINGING VOICE SYNTHESIS
C.I DETAILED ARCHITECTURE OF NANSY-SVS
Auto-regressive
Linguistic Feature
Linguistic Encader
Phoneme Seq.
Phoneme Encader
Ling. Fest. Decoder
Linguistic Feature
Global Timbar Embedding
Singer-ID Encoder
Amplitude Predictor
P/Ap Amplitdd
NANSY++ Synthesizer
Wave
MiDi-pitch Encoder
Residual-
MIDI-pitch Seq.
Fg
Residual-F,
Residual- Encader
MIDI-pich Seq.
Timbre
Auto regressive
Figure 14: Overview of NANSY-SVS.
NANSY-SVS generates linguistic features and residual-Fo in autoregressive manner using MIDI-pitch and phoneme sequence obtained from a musical score. In particular, to speed up an autoregressive inference, time-resolution, and then upsample them with the original time-resolution through an neural upsam- pler module. The dimensions of the input and output tensors of the model are shown in the table 9
first generate linguistic and residual-Fo features downsampled to 1/4
As shown Fig. [4| the NANSY-SVS model consists of five encoders, two decoder blocks, and an amplitude predictor. This section describes the detailed structure of each module in the order of encoder, decoder, and amplitude predictor.
Encoder Each encoder of NANSY-SVS is composed of combination of PreConv and ConvGLU modules. PreConv consists of LeakyReLU activation and Dropout followed by 1xl 1d convolu- tional layer. Unless otherwise stated, the dropout rate fixed 0.05. The ConvGLU module consists of Dropout, padding, 1d convolutional layer, and Gated Linear Unit. For the implemen- tation of casual and non-causal characteristics, used two different padding methods. For the
20

---

Table 9: The dimensions of the input and output tensors NANSY-SVS. B and T denotes batch
size and frame length, respectively.
Name
Shape
Input
midi-pitch sequence
[B. 1, T]
phoneme sequence
[B, 1, T]
linguisitic feature (downsampled)
[B, 128, T/4] residual feature (downsampled)
[B, 241, T/4] [B. 192, I]
singer embedding
Output
linguistic feature (downsampled)
[B, 128, T/4]
linguistic feature
[B. 128, T]
residual-Fo (downsampled)
[B, 241, T/4]
residual-Foo
[B, 241, TJ
P amplitude
[B. TJ
Ap amplitude
[B. 1, T]
causal module, padding was added to the front of the input sequence, and for the non non-causal mod- ule, the total padding length was divided in half and added to both sides. In order to optionally condition the singer ID embedding, a conditional layer normalization layer et al. (2020b) is added at the end of the module. Since the linguistic feature and residual-Fo are generated in an autoregressive manner, they have to be causally encoded in the encoding step. Therefore, as shown in Figure (c), Linguistic feature encoder and residual-Fo encoder are designed to go through a total of 10 causal ConvGLU blocks after two preconv and a point-wise convolutional layer. To widen the receptive field, we increase the dilation of the ConvGLU block to a power of 3. Similarly, the encoder that processes the phoneme sequence and MIDI-pitch sequence is designed as shown in Figure (d). Since phoneme and MIDI-pitch is categorical information, the embedding lookup table is added at the beginning of the module. In addition, a 1d convolutional layer with stride of 2 was added between ConvGLU modules match the time-resolution of encoded feature with the downsampled target feature. We used the NANSY++ backbone timbre embedding as the singer ID embedding for NANSY-SVS. We assume that this embedding already contains enough information about the speaker, so the singer-ID encoder is designed as simple fully-connected layer with ReLU activation.
output [BC.11
output
output [B,C) T/4]
outpot [
Singer-ID embedding
DonvGILL mej Reo, sull e-1 (1
output (R C')
(ostional)
Corv J-1 undi, prr, (Ulg)
X2
Convil (2, 2)
ConvOLU, d1
ConvGLU (1M3
x4
Dinpout
ConvCLUU_e_
ReLU
LeakyReLU
Convid (k, d.s)
ConvGLU [c.] [er), [er2,
Fullycconneted
Packting M -1)
Convid (1 1.1)
Convid
PreConv
Convid (1,1,
Drapout
PreCaev
[KC,7]
[n
Lookup
Input [B. C,1.
Imput [A
input [.T]
PreConv
ConvGLU_[eIne) (causal/ / non causall
Linguistic encoder / Residual-FO encader
Phoneme encoder MIDI-pptehh encoder
Singer-ID encoder
(+)
(b)
(c)
(d)
(e)
Figure 15: NANSY-SVS encoder architecture. h; d, and s of the Convle layer denotes kernel size, dilation, and stride, respectively.
21

---

Decoder Each decoder block of NANSY-SVS consists of casual and non- con-causal decoders and up- samplers as shown in Figure (d). Both the casual and non-causal decoders (Figure (a, b)) consist of 10 ConvGLU blocks and the last convolutional layer after PreConv module, and only the causality of ConyGLU is different. Singer-II embedding is input as a condition to all ConvGLU blocks, and is reflected through the conditional layer norm layer. The upsampler T con- sists of the nearest upsample layers following the ConvGLU block to upsample the time-resolution of the input by 4 times. In the linguistic feature decoder block, the output of [Ling., MIDI-pitch, phoneme} j encoder and singer-ID embedding are input to the causal decoder, and the output of the phoneme encoder and singer-ID embedding are input to the non-causal decoder. Similarly, in the residual-Fo decoder block, the output of the (Residual-Fo. phoneme, MIDI-pitch) encoder and singer embedding are input to the causal decoder, and the output of the MIDI-pitch encoder and singer embedding are input to the non-causal decoder. Finally, the model is trained so that the sum of the causal decoder and the non-causal decoder equals to the downsampled target feature, and the sum of the outputs of the two upsamplers equals to the original time-resolution target feature.
oulput (
output ()
output (R.C.7)
output 2 C.
(L 1,11
(onvid (1,19
Convid
DonvOLUL_cJ
Lel.y
x2
Joa dulli clM
X2
Conwtd (1, 1,
Upsampler
Upsampler
ConvCLLL [ع [kx3 (M
ComvCll Jon and
ConvCilinnee (h. dull d100
output C.
ConvGLU.c.J
ComvLLU.
Upsample (2)
ConvoLil_ne_ (k3 (k3 anll c0000
Causal decader
Non-causal decoder
PreComy
PreComv
Singer-D embediing
Singer-D embedding
PreComv
Input [B.C.7/4
inpueunc
imput (1))
input [
ieput [.
Causel decoder
Non-causel decoder
Upsampler
Decoder block
(a)
(b)
(c)
(d)
Figure 16: NANSY-SVS decoder architecture
Amplitude Predictor The amplitude predictor has the same structure as Figure 8-(b) Non-causal decoder, except that singer embedding is not used as an conditional input. The predicted linguistic feature and residual-Fo one of the analysis features of the NANSY++ backbone, are predicted. For training stability, the amplitude predictor was trained independently using a stop gradient layer instead of training with the entire network.
concatenated and used input, and P amplitude and Ap amplitude,
C.2 EVALUATION ON PITCH ESTIMATION AND VUV DECISION
For quantitative evaluation, we measured the Fo RMSE and voiced/unvoiced decision error rate between the ground truth and the generated singing voice. To extract F, from singing voice, we used used (Takin & Kleijn| 1995) pitch tracking algorithm. The evaluation results are shown in Table When all the training data were used, both models achieved similar performance. However, as the volume of training size was set to be smaller, Fo RMSE and VUV error rate of Diffsinger got degenerated significantly while those of NANSY-SVS did not. This demonstrates that NANSY-SVS faithfully reflects the input musical score condition even when trained with the small amount of data.
10
D VOICE DESIGNING
D.1 DETAILED ARCHITECTURE OF NANSY-VOD
Fig. 17 presents the detailed structures of the flow networks used in NANSY-VOD. Each flow layer conducts three processing steps; (i) normalization of input with learnable parameters (ActNorm),
22

---

Table 10: Fo and VUV difference between ground-truth and generated singing voice.
MODEL
F6 RMSE (cent)
VUV error rate (%)
BL / Ours (80)
233.69 99.118
12.93 / 5.67
BL Ours (160)
/ 101.46
9.05 / 4.95
BL / Ours (320)
125.97 94.96
7.39 / 5.35
BL / Ours (7ul))
97.96 / 96.90
4.79 / 4.89
NLL
NLL
Leyotlorm
Layer
X26
Layettern
Getli
Cender
Liver CC C.
Cendir
Tandon andeees
Destruped Idkams 1208
FO statasics [X]]]
Timbre embeoding [R) 192)
Eauelycores
Condition (B.C] Input (X
Timbre tokions
CondMLP
Timbre embedding flow
Timbre tokens flow
(+)
(c)
Figure 17: Flow architecture of NANSY-VO..
(ii) shuffling input across channel dimension (ShuffleLayer), and (iii) applying invertible bijective function (AffineCouplingLayye
ActNorm normalizes data by using the process y (z-8) where and y are learnable parameters of the same dimension as the input dimension. 3 and y are initialized such that y has zero mean and unit variance given initial training batch. ShuffleLayer swaps the first half of the channels with the other half. offineCopllngglayy. transforms the half of the input channels while keeping the other part follows:
(a, w) = CondMLP(xs)
(2)
V/ = a ② Fa +
(3)
yb =
(4)
where va and Tb denotes the first and second part of the input respectively.
NANSY-VOD consists of three flow networks; Fo flow, timber embedding flow, and timbre tokens flow. Fo flow employs 8 flow layers and uses age and gender as conditional variables. Similarly, time embedding flow is constructed of 12 flow layers conditioned by age, gender, and Fo statistics. Timbre tokens flow consists of 16 flows layers with age, gender, Fo statistics, global timbre embed- ding and token index as conditional terms. Timbre tokens are reshaped to form multiple individual data points, passed to 16 layers of timbre tokens flow, and processed independently. To distinguish timbre tokens, the corresponding token index used as an additional conditional input.
D.2 AGE CONTROL.
We note that gender labels estimated by SPNet show different trend depending on age. We anno- tated continuous gender labels of the 320 utterances using SPNet and report the result in Fig. [8]] The gender labels of children under age of 10 are clustered around [-5, 5] while the other gender labels are scattered on both sides. From this, instead of adjusting an age value only, we devise an age control algorithm for NANSY-VOD taking into account the dependency between age and con- tinuous gender. When the age value is adjusted from under 10 to over 10 years (i.e., from child voice to a mature voice), the gender value is multiplied by 8. When the age value is edited from over 10 to under 10 years (i.e.. from mature voice to child voice), the gender value is divided by 8. In other cases, we simply control the age value only. In our preliminary experiments, we empirically verified that this algorithm significantly improves the age editing performance of NANSY-VOD.
23

---

Continuous gender
-10
10
20
30
40
50
60
70
Age
Figure 18: Estimated age and gender distribution on test set samples. The gender distribution ap- pears to have bi-modal distribution for the people who are older than 10 years old, while children who are younger than 10 years old appears have uni-modal distribution around 0.
D.3 INFERENCE OVERVIEW
Urgussic Festure
Amploude
Mudian
Medaa!
Gobal Timbre Embodding
Fu Std.
Global Timbro Embebdinng
R Std!
Synthesber
Ware
Source Age/Gender
Age/Genner
Velua Arst.ua
Grossion Neise
"ole
Intent Vaniblee
Figure Overview of inference procedure of NANSY-VOD.
Fig. 19 demonstrates the inference procedure of NANSY-VOD. To edit the voice attributes of source audio, NANSY-VOD transforms FO statistics, global timbre embedding, and timbre tokens into latent variables, and converts them back to analysis features. Note that NANSY-VOD uses source speaker's age and gender for the forward process, and the target age and gender for the inverse process. To generate new voice identity, latent variables sampled from the normal distribution and transformed into analysis features according to the target age and gender.
D.4 SPNET
SPNet is neural network for estimating speech parameters such age and gender given speech. SPNet employs the same architecture as ECAPA-TDNN (Desplanques et al.) 2020) and stacks projection layer on top of it. The last layer projects 192 dimension dimension, and each output is used to estimate age and gender. Mean absolute error and soft margin loss are employed as an objective function for age and gender estimation, respectively. AI-Hub conversation datasets were used for both training and evaluation. We initialized SPNet with the pretrained ECAPA-TDNI weights and trained it using the AdamW optimizer (Loshchilov & Hutter) 2017) learning rate 2 x 10-4 for about 33K iterations. Each audio data randomly cropped one second and used to construct a mini batch size 256. At test time, we sampled 320 utterances from various age and gender groups, and estimated speakers' age and gender using SPNet. The average absolute error of age estimation was 4.095 and the average accuracy of gender estimation was 93.75%. Since voices of pre-puuescent children may be gender distinct, also evaluated gender estimates for people over the age of 10. In this case, the average accuracy recorded 99.179.. 17%.
24


Reviews:
[{"review_id": "jrHWyDy50ux", "reviewer": "Reviewer_vW43", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- Authors propose a unified framework to learn many voice synthesis tasks. The unified framework includes task-independent self-supervised training of a backbone network, which does not require any labelled data, and then task specific training with labelled data.\n- Authors claim that this unified approach offers advantages such as controllability of synthesized speech, data efficiency for downstream tasks, and fast training convergence — all this without sacrificing the quality of synthesized speech.\n- Authors assume that the most of the voice synthesis tasks can be defined synthesizing and controlling our aspects of voice, that is, pitch, amplitude, linguistic, and timbre.\n- For estimating pitch, in an unsupervised fashion, authors use the property of CQT input representation, and the transforming the representation appropriately to get a known shift in the fundamental frequency value. While this will bias learning of fundamental frequency like value, it is not clear how authors obtain absolute fundamental frequency. This method is inspired by “SPICE: Self-supervised Pitch Estimation”.\n- For getting linguistic features, authors propose to use a contrastive learning approach where they modify the signals in a way to keep the linguistic information intact, while changing other properties of the signal — and use this property learn features which maximizes similarity if the source audio (i.e. before transformations) is the same i.e positive samples in the contrastive learning approach are two different transformations of the same signal and negative samples are the transformations of two different signals. Negative samples clearly have different linguistic information, but it is not clear that they will share the same non-linguistic features — maybe that is not necessary for this approach to work. Authors extract wav2vec features from transformed signal, which is transformed via another learnable network. I am curious about the importance of this contrastive learning approach vs directly using wav2vec features.\n- Authors propose learnable vector encoding of input signal — one global vector and then time varying tokens — to represent timbre. It is not clear why these learnable encodings will necessarily encode timbre, and will necessarily be disentangled from pitch and linguistic representation.\n- Authors propose to use Parallel WaveGan trained via reconstruction loss  on mel-spectrogram, and adversarial and feature matching loss to train the synthesizer network.\n- Experiments:\n    - Authors shows that replacing fundamental frequency extracted from the analysis network by the fundamental frequency extracted from four other approaches, and show that the synthezised audio using their synthesizer is assessed to match the original signal before doing the f0 replacement. It is not clear why this implies that their analyser learns better f0 — it is totally possible and likely that even if their analyser is not learning the correct f0, but since the representation is trained end-to-end along with synthesizer, reconstructions with this representation are closer to original, and using a more correct (or even a different f0) will throw the model off.\n    - Authors show better reconstruction quality than a mel2wav architecture namely HiFiGAN. Since the input representations are different, specifically authors choice of input is much more rich than mel-spectrogram used in mel2wav setup. It is not clear why having better reconstruction is a good metric to measure in this case.\n    - Authors show clear improvement in zero-shot voice conversion, when compared with the baseline.\n    - Authors show better performance than the baseline on TTS and Singing voice synthesis. The baseline for full data seem far from real audio, which probably points to the baseline being weaker than SOTA TTS.\n    - Authors show good results on Voice Designing.", "summary_of_the_review": "Overall, the authors should reconsider claims about disentangled representation learning. Should consider doing ablations for various choices, and improve clarity of the writing.", "strength_and_weaknesses": "- Overall:\n    - Authors claim about disentangled representation learning of f0, timbre, linguistic information might be misleading. I wonder if authors will consider changing the wording to showcase what design choices they make that promote learning disentangled representation, and they verify experimentally that some of it is achieved.\n    - Experimental results on downstream tasks seems very good. The impact of the work can be maximized by making sure that the baseline performance is replicated from original work e.g. baseline TTS on full data, seems to be very far from real data MOS, which does not reflect the SOTA on TTS tasks.\n\nStrengths:\n- Extensive evaluation\n- Impressive Results\n\nWeakness:\n- Not so great scientific rigor\n- Not so-great choice of baselines\n- Potentially misleading claims or just confusing to understand what is claimed vs what is expected.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ih_lXdQFsEh", "reviewer": "Reviewer_Hzqp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a unified framework for human voice synthesis based on neural analysis and synthesis modules. The analysis part extracts disentangled and controllable voice features based on domain knowledge (pitch, periodicity/aperiodicity and timbre). Each of them is trained independently with a dedicated neural analysis module. The timbre analysis module divided the feature into a global timbre embedding vector and time-varying timbre tokens.  The synthesis part take the voice features and generates waveforms with frame-level and sample-level synthesis modules. The framework is evaluated in terms of F0 estimation and reconstruction error as a vocoder, showing superior results.  The framework is applied to four voice synthesis tasks including voice conversion, TTS, singing voice synthesis, and voice design, along with additional encoder modules. In each task, the proposed framework outperform previous state-of-the-arts. \n\n\n", "summary_of_the_review": "This paper proposes a flexible voice synthesis framework with high-performance. The framework include robust F0 estimation and high-quality reconstruction as a vocoder.  The flexible structure was applied to voice conversion, TTS, singing voice synthesis and new voice design. I believe that the framework is highly well designed and engineered. \n\nOne main issue of this paper is the writing structure. The main body is a summary of long manuscripts on \"a large system\" and all details are moved to the appendix section. I feel like it would be better for the authors to submit this paper to a journal which allows many pages. \n\nIn addition, the novel part such as time-varying timbre embedding was not thoroughly validated, which might be probably due to the space issue. \n\nLastly, the high-performance of the framework is attributed to not only the model architecture but also the large-scale training data. This should be clearly indicated when compared to other models. \n ", "strength_and_weaknesses": "Strengths\n- The proposed framework is highly modular and flexible, being plugged in various voice synthesis tasks\n- The framework is interpretable as it is designed to be similar to DSP-based parametric vocoders. \n- They achieve impressive results in all tasks, outperforming compared models.   \n\nWeakness\n- The paper includes too much content. Due to the space issue, all details are moved to the appendix.  \n- Although the proposed framework is well-structured, it is extended from the authors' previous work (NANCY) and many new components are adopted from other previous work (self-supervised pitch estimation, linguistic representation, parallel WaveGAN, and so on). \n- While the framework achieves better performances than previous work in many tasks, the experiment condition is not completely fair because the NANCY framework was pretrained with a large-scale dataset (10,571 hours from 6,176 speakers and 624 singers).  \n- The time-varying timbre embedding is a great idea but it was not validated through an ablation study. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EHSyz7O-UG", "reviewer": "Reviewer_U62B", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed NANSY++, a unified framework of synthesizing and manipulating voice signals from analysis features. The backbone network of NANSY++ is trained in a self-supervised way. After the backbone network is pre-trained, different voice applications can be adapted by modeling the analysis features required for each task. Experiments show that the proposed NANSY++ framework has the advantages of controllability, data efficiency, fast convergence, and high quality synthesis.", "summary_of_the_review": "The proposed unified voice synthesis framework NANSY++ achieved impressive performance and can serve as a general speech pre-processing and pre-training pipeline.\n\n\n=================\n\nUpdates after reading the author's rebuttal: the author addressed my concerns well, so I updated my rating to 8.", "strength_and_weaknesses": "It's impressive that NANSY++ can use one unified framework to handle four different speech tasks while achieving good performance in audio quality, data efficiency and controllability. The framework is well designed in a modularized way and can disentangle various speech representations. One drawback of this paper, however, is limited ablation study. Since the proposed NANSY++ framework requires a variety of modules and augmentations, it's hard to know what are the core factors that contribute to the model performance.\n\nSeveral detailed questions:\n\nIn section 2.2, for the linguistic feature disentanglement, what's the exact equations of breathiness perturbation?\n\nIn section 2.3, are the global and time-varying timbre embeddings also trained in a self-supervised way or trained jointly with the waveform reconstruction? If they are jointly trained, how to guarantee the time-varying timbre embeddings don't contain other information such as pitch and linguistic?\n\nIn section 4.2.2, what's the problem definition here for zero-shot TTS? Does the mean the model trained on TTS task can be adapted to a new dataset or the pre-trained backbone model can be used directly for TTS task?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "AnxOTI4lJR3", "reviewer": "Reviewer_D823", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a learned analyzer and synthesizer for speech that can be trained end-to-end, and demonstrates their utility in several speech and singing synthesis tasks: voice conversion to match a target individual, voice conversion to match target characteristics, text to speech, and singing voice synthesis. In listening tests for the targeted voice conversion, NANSY++ exceeded YourTTS  by 0.56 MOS points (from 1-5 range) in quality and 0.38 points (1-5 range) in speaker similarity. For TTS, compared with Glow-TTS, the proposed approach performs better in quality and character error rate of a pre-trained recognizer at various amounts of training data from a single speaker (5 mins, 10 mins, 30 mins, and 30 hours). For singing voice synthesis, the proposed approach out performed diffsinger in terms of subjective quality with various amounts of training data. And for voice conversion to match target characteristics, the proposed approach was able to generate 320 speakers that achieved an objective voice diversity similar to that of 240 actual different speakers.", "summary_of_the_review": "End-to-end trainable voice analysis-synthesis pipeline that cleverly separates various voice characteristics through relative contrasts. Experiments show that it is useful and data efficient in several different tasks. Minor issues with clarity of captions and with certain claims being slightly overstated.", "strength_and_weaknesses": "Strengths:\n* Well executed experiments: the experiments are all clearly described, well performed, and test key claims of the paper\n* Approach works well. The results of those experiments show that the proposed method does provide superior performance to the various baseline approaches.\n* Model provides general utility. As demonstrated by the variety of experiments conducted, the proposed model is useful in a large number of situations without requiring a large amount of data or a large amount of time to retrain. For example, for TTS, between 30 minutes and 30 hours of data, the model improves in MOS from 3.64 to 4.07 and in CER from 2.20 to 1.68%. These are meaningful improvements, but the model trained on 30 minutes is still performing quite well and fairly close to the model trained on 30 hours.\n* General clarity of presentation. The paper is generally well presented, except for the issue of certain claims being slightly over stated mentioned below under weaknesses.\n\nWeaknesses:\n* The captions of the figures and tables are too brief and should be expanded so that figures are clearly interpretable by themselves without referring to the text.\n* Certain claims are slightly overstated, although these claims are also not especially necessary for the work to have impact. In particular, I am referring to claims that the model is disentangling various voice characteristics and that it is self-supervised. While both of these may be technically true, it is thanks to careful inductive biases baked into the model architecture. Thus it is not especially surprising that the model has these characteristics, but it's not like they just popped out of the data. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster