Interspeech 2022
18-22 September 2022, Incheon, Korea

MSR-NV: Neural Vocoder Using Multiple Sampling Rates
Kentaro Mitsui, Kei Sawada
rinna Co., Ltd., Japankemits@rinna.co.jp, keisawada@rinna.co.jp

Abstract

regard to speech, its content and individuality are concentrated
in the low frequency range. Therefore, sampling rates of 16,
22.05, and 24 kHz are often used in research on NVs. In an
environment where only synthetic speech is heard, it does notmatter even if the human audible range (20–20,000 Hz) is notentirely covered. However, in situations where synthetic speechis used for conversing with a human, or where it is played simultaneously with the sound of musical instruments in music, thesynthetic speech may sound muffled compared to other sounds.
Some studies (Full-band LPCNet [20] and PeriodNet [21]) havetackled this problem by generating 48 kHz waveforms. However, in general, the higher the sampling rate, the more difficult it is to generate high-quality waveforms because of theneed to model long-term dependencies. Additionally, because
conventional NVs target a specific sampling rate, speech withsampling rates lower than that of the target cannot be used fortraining. This is because even if we upsample a waveform withsuch low sampling rates, high-frequency components cannot berecovered. Although large speech corpora such as the LJSpeechdataset 1 and LibriTTS [22] have been recently released, 44.1 or
48 kHz datasets are still limited.
For these reasons, a method that: 1) can properly model
speech with high sampling rates, and 2) also use speech witha sampling rate lower than that of the target for training, isrequired. In this study, we propose multiple sampling rate(MSR) -NV as a method with the aforementioned characteristics. MSR-NV can generate speech waveforms step-by-step,
starting from a low sampling rate. More specifically, after generating a waveform with a certain sampling rate, sinc interpolation is performed to upsample the waveform, and a neuralnetwork is used to predict the residual high-frequency components to generate a waveform with a higher sampling rate. This
would allow different networks to capture the features containedin each frequency band and efficiently model waveforms withhigh sampling rates. Additionally, MSR-NV enables us to train
a part of the model using speech with a low sampling rate. As aresult, it is expected that speech with sampling rates that could
not be used in the past can be used together for training, whichleads to the realization of a more general-purpose NV.
We conducted experimental evaluations using a model
structure based on Parallel WaveGAN (PWG) [12] to demonstrate the effectiveness of the proposed method. First, we compared the subjective quality of the speech generated at 16, 24,
and 48 kHz using the proposed method with that of the speech
generated using the baseline model that was trained separately
at these sampling rates. To investigate the data efficiency ofthe proposed method, we evaluated the quality of the synthetic
speech when the amount of training data was varied from 1 min
to 8 h. Finally, we confirmed that the quality of synthetic speech
can be improved using speech with lower sampling rates in conjunction with the original training data.

The development of neural vocoders (NVs) has resulted in thehigh-quality and fast generation of waveforms. However, conventional NVs target a single sampling rate and require retraining when applied to different sampling rates. A suitablesampling rate varies from application to application due to thetrade-off between speech quality and generation speed. In thisstudy, we propose a method to handle multiple sampling rates
in a single NV, called the MSR-NV. By generating waveforms
step-by-step starting from a low sampling rate, MSR-NV canefficiently learn the characteristics of each frequency band andsynthesize high-quality speech at multiple sampling rates. It canbe regarded as an extension of the previously proposed NVs,
and in this study, we extend the structure of Parallel WaveGAN(PWG). Experimental evaluation results demonstrate that the
proposed method achieves remarkably higher subjective quality than the original PWG trained separately at 16, 24, and 48
kHz, without increasing the inference time. We also show that
MSR-NV can leverage speech with lower sampling rates to further improve the quality of the synthetic speech.
Index Terms: neural vocoder, speech synthesis, sampling rate,
generative adversarial networks, Parallel WaveGAN

1. Introduction
In text-to-speech synthesis, singing voice synthesis, and music synthesis, studies to improve the quality of vocoders, which
generate waveforms from acoustic features, have been extensively conducted. Neural vocoders (NVs), which use neuralnetworks to generate waveforms, have greatly improved thequality of synthetic audio. WaveNet [1, 2], a representative ofautoregressive (AR) NVs, achieved a significantly higher quality than conventional signal processing-based vocoders [3, 4]
by predicting waveform samples individually. Although AR
NVs can generate high-quality audio because they can use previous prediction results, they suffer from a slow generation
speed. To address this issue, fast and high-quality waveformgeneration using non-autoregressive (non-AR) NVs has been
actively studied. Various generative models have been usedin this area, including inverse autoregressive flow (IAF) [5]
used in Parallel WaveNet [6] and ClariNet [7], generative flow
(Glow) [8] used in WaveGlow [9], the generative adversarial
network (GAN) [10] used in several NVs [11, 12, 13, 14], andthe denoising diffusion probabilistic model (DDPM) [15] usedin WaveGrad [16] and DiffWave [17]. Moreover, several methods incorporating signal processing insights have been proposed, including the neural source-filter model [18] and LPCNet [19].
The sampling rate of the waveform plays a key role in waveform generation. It represents the resolution in the time domain
when waveforms are being handled on a computer. The higher
the sampling rate, the more accurately the original waveform
can be expressed, but the larger will be the amount of data. With

Copyright © 2022 ISCA

1 https://keithito.com/LJ-Speech-Dataset/

798

10.21437/Interspeech.2022-295

Generated waveform

Generated waveform 𝐱 𝑖Generator 𝐺𝑖

𝐡𝐼

UpsamplehighHigh frequency component 𝐱 𝑖𝐡2Neural network 𝜙𝑖𝐱2

UpsampleLow frequency component 𝐱 𝑖low𝐡1

Upsample

Waveform generated in previous stage 𝐱 𝑖−1

𝐱𝐼

Local conditioning feature 𝐡𝑖

𝐱1

Generator 𝐺𝐼

Generator 𝐺2

Upsample

Generator 𝐺1

Local conditioning
feature 𝐡Noisesequence 𝐳

Real or Fake

Target waveform

Discriminator 𝐷𝐼
Downsample
Real or Fake

Discriminator 𝐷2
Real or Fake

Downsample

Discriminator 𝐷1

Figure 1: Algorithm of generating waveforms of multiple sampling rates step-by-step.

Figure 2: Parallel WaveGAN-based model structure extended
using the proposed method.

2. Related works

following procedure, we can sequentially generate a waveformwith a higher sampling rate starting from a low sampling rate:

Obtaining a high-resolution output is a common challenge not
only for speech waveform generation, but also for image generation. Progressive growing GAN [23] enables the generationof images with unprecedented 1024×1024 pixels by the gradual addition of layers corresponding to higher resolutions. The
later published StyleGAN [24] enables the generation of highresolution images without changing the topology of the network by preparing layers for multiple resolutions in advance.
Each layer of the network can represent features corresponding
to different resolutions using these methods, which can handle
multiple resolutions in a stepwise manner.
Several methods have been proposed for speech waveformgeneration to handle multiple sampling rates. MelGAN [11]and HiFi-GAN [14] are methods that can predict waveformby repeatedly upsampling and transforming features. HiFiGAN achieves a quality comparable to natural speech at 22.05
kHz. VocGAN [13] has a generator similar to that of MelGAN, but generates and evaluates waveforms with ×1/n (n =2, 4, 8, 16) sampling rates. However, while the proposed
method directly upsamples the waveform and predicts the residual high-frequency components, VocGAN upsamples the features; thus, when a waveform is viewed at multiple sampling
rates, the residual structure is not explicitly used. Additionally, when a high sampling rate (e.g., 44.1, 48 kHz) that covers
the entire human audible range is targeted, it is difficult to use
speech data with lower sampling rates (e.g., 16, 22.05 kHz) for
training. Even if we upsample them, high-frequency components cannot be recovered.

1. Upsample xi−1 to sampling rate fi to approximate xlowi
2. Based on the upsampled waveform xlow
and the coni
ditioning feature hi (e.g. upsampled melspectrogram),
predict xhigh= ϕi (xlowi , hi ) using a neural network ϕii
3. Calculate the sum of 1 and 2: xi = xlow
+ xhighi
i
The module that performs the abovementioned procedure is referred to as Generator Gi . Fig. 1 illustrates these steps. Regarding step 1, transposed convolution is often used for upsamplingin NVs [11, 13, 14]. However, this requires that fi is divisible by fi−1 and may include high-frequency components in therange [fi−1 /2, fi /2] that were not included before upsampling.
For these reasons, the proposed method uses upsampling basedon sinc interpolation.
Then, we consider the case where i = 1. When generating the waveform x1 of the lowest sampling rate f1 , the previous waveform x0 does not exist. Thus, the function ϕ1 predicts
x1 = ϕ1 (h1 ) with only the conditioning feature h1 as the input.
In the proposed method, f1 can be set arbitrarily. According toOura et al. [25] and Hono et al. [21], a sinusoidal input corresponding to the fundamental frequency of speech is effective in
predicting the periodic component of speech. Therefore, we set
f1 such that the fundamental frequency is included in [0, f1 /2]
to make x1 similar to the sine wave corresponding to the fundamental frequency. We expect that x1 will facilitate waveformgeneration in the subsequent stages.

3. MSR-NV

3.2. Model details

3.1. Sequential waveform generation of multiple sampling
rates

The proposed method described in section 3.1 can be applied to
NVs of various structures and enables training at high sampling
rates, which is difficult with conventional NVs. In the case of
I = 1, the model structure matches that of a conventional NV,
and in the case of I > 1, it can be regarded as an extension
using the proposed method. We used a model structure basedon PWG [12]. Although PWG has undergone several improvements since it was first introduced [26, 27, 28, 29, 30], we used
the original PWG to verify the effectiveness of the proposed
method with the simplest model structure. A conceptual model
is depicted in Fig. 2. As the function ϕi described in section 3.1,
we use the same structure as the generator of PWG, that is, a

In this section, we describe the procedure for generating
speech waveforms at multiple sampling rates using the proposed method. Let f1 < · · · < fi < · · · < fI be the sequenceof the sampling rates to be generated.
First, we consider i ≥ 2, where we predict the waveform xi
with a sampling rate fi from waveform xi−1 with a sampling
rate fi−1 . From the linearity of the Fourier transform, xi canbe expressed as the sum of a waveform xlow
with a componenti
at frequency [0, fi−1 /2] and a waveform xhigh
with a compoi
nent at frequency [fi−1 /2, fi /2]. Therefore, by repeating the799

Table 1: Results of MOS evaluation on closeness to recorded
speech quality with 95% confidence intervals.

non-causal WaveNet. To predict xhigh= ϕi (xlowi , hi ) with ai
WaveNet-based structure, we must match the temporal resoluand hi . Thus, we obtain hi by properly upsamplingtion of xlowithe original conditioning feature h with sinc interpolation. In
the case of i = 1, we follow PWG and predict x1 = ϕ1 (z, h1 )using white noise z as input, in addition to the conditioning feature h1 .
The proposed method generates I waveforms from x1 to
xI . Therefore, we use I discriminators {Di }Ii=1 to identify
whether the corresponding waveform is real or fake. All thediscriminators have identical structure to that of PWG. A multiscale discriminator (MSD) [31] has also been proposed as astructure to identify downsampled waveforms. However, MSD
loses high-frequency components owing to low-pass processing associated with average pooling. Thus, we downsampled
the natural speech in advance using an anti-aliasing filter andsinc interpolation, and used it as a target at each sampling rate.
We simply extend the loss function used in PWG [12] tomultiple sampling rates. For each sampling rate fi , we add
multi-resolution short time Fourier transform (MR-STFT) loss
Laux (Gi ) to adversarial loss Ladv (Gi , Di ) weighted by λadv ,and use the sum of these as the loss function LG (G, D) for theentire generator. For the discriminator, we use the sum of thelosses Ldis (Gi , Di ) for the discrimination results of Di as theloss function LD (G, D) for the entire discriminator. The above
can be expressed as follows:I
X
LG (G, D) ={Laux (Gi ) + λadv Ladv (Gi , Di )} (1)LD (G, D) =i=1

Ldis (Gi , Di )

Ref

PWG

MSR-PWG

single/16k
single/24k
single/48k

2.48±0.16
3.97±0.15
4.43±0.14

1.50±0.11
2.21±0.15
1.95±0.15

2.26±0.14
3.47±0.15
3.80±0.16

multi/16k
multi/24k
multi/48k

3.11±0.14
4.25±0.13
4.55±0.10

2.41±0.13
2.50±0.15
1.84±0.12

2.35±0.12
3.06±0.15
3.29±0.15

included in [0, f1 /2] in most of the frames. Hereafter, we denote the experimental conditions for generating 48 kHz speechusing the proposed method as MSR-PWG-48k. All {ϕi }7i=1
have identical structures of a ten-layer, one-stack non-causal
WaveNet with dilation set to 1, 2, 4, . . . , 512. Following theexperimental conditions of PWG, the number of channels forthe residual block and skip connection was set to 64, and thesize of the convolutional filter was set to three. Upsampling of
waveforms and conditioning features was conducted in a differentiable manner using torchaudio [32]. All discriminators wereconstructed with the same structure as that of the PWG, thatis, ten-layer non-causal dilated convolutions with a leaky ReLU
activation function (α = 0.2).
The parameters of the MR-STFT loss were set to different values for each sampling rate. For f7 = 48 kHz, we
set the frame length to {2048, 4096, 1024} points, the window length to {1200, 2400, 480} points, and the frame shift to{240, 480, 100} points. For {fi }6i=1 , these parameters were
multiplied by fi /f7 . The weight of adversarial loss λadv was
set to 1.0. A mini-batch was constructed by randomly clipping speech (0.5 s) from eight utterances for each training step.
Training was conducted for 400,000 steps using RAdam optimizer (ϵ = 10−6 ). Only the generators were optimized in
the first 200,000 steps, and the generators and discriminators
were jointly optimized in the following 200,000 steps. The initial learning rates were set to 10−3 for both the generator anddiscriminator, and they were reduced by a factor of 0.5, after
300,000 steps of training.
As baseline models, we trained the original PWG with 16,
24, and 48 kHz speech waveforms, respectively (hereinafter referred to as PWG-{16,24,48}k)2 . To extract log mel spectrograms, the frame and window lengths were set to 512, 1,024,
and 2,048 points for PWG-16k, PWG-24k, and PWG-48k, respectively. The frame shift was set to 5 ms for all settings. For
PWG-16k and PWG-24k, the parameters of the MR-STFT loss
are the same as in the previous study [12], and they were doubled for PWG-48k.

i=1

I
X

Dataset(2)

When we use speech with a sampling rate fJ (J < I) lower
than fI for training, we can train a part of the network bychanging the range of summation in Eqs. (1) and (2) fromi = 1, . . . , I to i = 1, . . . , J.

4. Experiments
4.1. Experimental conditions
Experiments were conducted in two settings: single-speakerand multi-speaker. In the single-speaker experiment, a totalof 14,375 utterances (approximately 8 h) uttered in a normal
speaking style by a female Japanese speaker were used. For thedevelopment and evaluation sets, respectively, 300 utteranceswere randomly selected, and the remaining 13,775 utteranceswere used as the training set. In the multi-speaker experiment,a total of 31,936 utterances (approximately 25 h) by 26 Japanese
speakers (18 female and 8 male), including three speaking styles(normal, happy, and sad) were used. A total of 520 utterances,
20 from each speaker, were used for the development and evaluation sets, respectively, and the remaining 30,896 utteranceswere used for the training set. All the experiments were conducted using 48 kHz/16 bit speech signals. They were trimmed
so that the silent interval before and after the speech was approximately 200 ms. Eighty-dimensional log mel spectrogramswith bands in the range 80–7,600 Hz were used as the conditioning feature h described in section 3.2. They were extractedwith a frame and window length of 2,048 points (approximately
42.7 ms) and a frame shift of 240 points (5 ms) and were normalized to zero mean and unit variance.
The sampling rates handled using the proposed method
were set to I = 7 and {fi }7i=1 = {1, 2, 4, 8, 16, 24, 48} kHz.
f1 was set to 1 kHz so that the fundamental frequency would be

4.2. Evaluation
4.2.1. Comparison to baseline in terms of quality and speed
A mean opinion score (MOS) test was conducted to evaluate
the subjective quality of synthetic speech. In both of the singlespeaker and multi-speaker experiments, each rater evaluated ten
sets of 9 speech samples, for a total of 90 samples, where each
set consisted of a combination of three sampling rates (16 kHz,
24 kHz, and 48 kHz) and three methods (reference: Ref, conventional method: PWG, and proposed method: MSR-PWG)32 https://github.com/kan-bayashi/ParallelWaveGAN was used.
3 Speech samples are available at the following URL: https://rinnakk.github.io/research/publications/MSR-NV.

800

Table 3: Comparison of MOS for various training data amountwith 95% confidence intervals.

Table 2: Real-time factor of inference averaged over 300 utterances.
Method

16 kHz24 kHz48 kHzTraining data amountPWG
MSR-PWG

0.027
0.027

0.039
0.042

0.074
0.068

1 min
3 min
5 min
10 min30 min8 h (full data)

Speech samples were evaluated in terms of closeness to thequality of the 48 kHz reference samples using a five-point scalefrom 1 (very far) to 5 (very close). The raters should be able todistinguish between 16, 24, and 48 kHz speech samples; however, depending on the playback equipment and the subject’s
hearing, it may not be possible to distinguish between these different sampling rates. Thus, we conducted an internal preliminary test and asked twenty raters who were able to distinguishbetween them to participate in the evaluation.
The results are presented in Table 1. Note that the scores of
PWG are lower than those of the original paper [12] owing to
the differences in the experimental condition and evaluation criterion. In the single-speaker experiment, MSR-PWG achieved
higher scores than PWG at any sampling rate. The score of
PWG-48k was lower than that of PWG-24k, even though it
was closer to the reference speech in terms of the sampling rate.
This is because PWG had difficulty generating waveforms witha high sampling rate, resulting in quality degradation. However, MSR-PWG achieved a high score even at 48 kHz, which
verifies the effectiveness of stepwise waveform generation starting from a low sampling rate. In the multi-speaker setting, thescores of MSR-PWG were comparable to PWG at 16 kHz andsignificantly higher at 24 kHz and 48 kHz. However, there wasa gap between the scores of Ref and those of MSR-PWG. In
particular, the speech quality was degraded for male speakers,
whose amount of data was relatively small in this experiment.
Improving the performance of multi-speaker multi-style modeling is a future challenge for the proposed method.
In terms of model size, the numbers of parameters were
1.44M for PWG and 3.06M for MSR-PWG. When we expanded PWG to 70 layers to align the number of its parameterswith MSR-PWG, the training and inference time increased significantly without any quality improvement. Thus, we conclude
that not the increased number of parameters but the proposed
structure improved the speech quality.
We also measured the training and inference speed of thebaseline and proposed methods using NVIDIA Tesla P40. The
training required 103 h for PWG-16k, 153 h for PWG-24k,
252 h for PWG-48k, and 132 h for MSR-PWG. Although theproposed method handled up to 48 kHz waveform generation, it
achieved high-quality synthesis with a training time shorter thanthat of PWG-24k. For inference, the real-time factor (RTF),
which is the time required to generate a waveform in one second, was measured, and the results obtained are presented inTable 2. Although the generator of the proposed method consists of 70 layers, compared to 30 in the baseline, the inferencetime did not show an increase. This is because the input lengthis proportional to fi , which is especially short in the early stagesof the model, and the computational cost is reduced.

MOS
1.49±0.13
3.07±0.15
3.43±0.16
3.50±0.15
3.55±0.16
3.75±0.16

Table 4: Comparison of MOS for different training sets with
95% confidence intervals.
Training set

MOS

1min
3×1min3min

1.76±0.14
2.67±0.15
3.32±0.17

was only 1 min, it greatly improved when 5 min of training data
was used. This result indicates that the proposed method cangenerate speech of adequate quality even with 5 min of training
data.
4.2.3. Use of speech data with low sampling rates
We investigated whether the quality could be improved using
16 and 24 kHz speech for training when there is little 48 kHz
speech available. The following three conditions were evaluated in the single-speaker setting using the MOS test similarly
to section 4.2.1: 1min: 1 min of 48 kHz speech was used fortraining, 3×1min: 1 min of different 16, 24, and 48 kHz speech,
for 3 min, was used for training, and 3min: 3 min of 48 kHz
speech was used for training. The results are presented in Table 4. 3×1min showed a significantly higher score than 1min,confirming that the quality of synthetic speech can be improved
using speech with a lower sampling rate for training. Although
we used only one minute each of 16 and 24 kHz speech here, thesynthesis quality of 48 kHz speech could be further improved byincreasing the amount of these lower sampling rate speech data.
However, because 16 and 24 kHz speech do not contain any
components above 8 and 12 kHz, respectively, only a part of the
network can be trained with these data. This is why the score of
3×1 min was lower than that of 3 min, which was trained using
the same amount of 48 kHz speech.

5. Conclusions
In this study, we proposed MSR-NV, a method to handle multiple sampling rates using a single neural vocoder. Experimental evaluations using a PWG-based structure demonstrated thatthe proposed method could generate high-quality waveforms at
multiple sampling rates, including 48 kHz, while maintaining afast generation speed. We also demonstrated the data efficiencyof the proposed method by varying the training data amount andevaluating the speech quality. In addition, we could improve
speech quality using 16 and 24 kHz speech when the amountof 48 kHz speech was limited. The proposed MSR-NV eliminates the need to re-train the model for different sampling rates,
which broadens the range of applications.
Future work includes verifying the effectiveness of the proposed method when model structures other than PWG are used.
It is also necessary to improve the speech quality for multiple
speakers and speaking styles to realize a more versatile vocoder.

4.2.2. Training data amount and synthesis quality
Six models were trained with single-speaker data of 1, 3, 5, 10,
30 min, and 8 h (full data). The quality of the 48 kHz synthetic
speech was evaluated using the MOS test in the same manner as
in section 4.2.1. The results are presented in Table 3. Although
the score was significantly low when the amount of training data801

6. References[19] J.-M. Valin and J. Skoglund, “LPCNet: Improving neural speechsynthesis through linear prediction,” in Proc. ICASSP, Brighton,
U.K., May 2019, pp. 5891–5895.

[1] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals,
A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu,“WaveNet: A generative model for raw audio,” arXiv preprint
arXiv:1609.03499, Sep. 2016.

[20] K. Matsubara, T. Okamoto, R. Takashima, T. Takiguchi, T. Toda,
Y. Shiga, and H. Kawai, “Full-Band LPCNet: A real-time neuralvocoder for 48 kHz audio with a CPU,” IEEE Access, vol. 9, pp.
94 923–94 933, Jun. 2021.

[2] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, andT. Toda, “Speaker-dependent WaveNet vocoder,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017, pp. 1118–1122.

[21] Y. Hono, S. Takaki, K. Hashimoto, K. Oura, Y. Nankaku, andK. Tokuda, “PeriodNet: A non-autoregressive waveform generation model with a structure separating periodic and aperiodic components,” in Proc. ICASSP, online, Jun. 2021, pp. 6049–6053.

[3] H. Kawahara, M. Morise, T. Takahashi, R. Nisimura, T. Irino,and H. Banno, “Tandem-STRAIGHT: A temporally stable power
spectral representation for periodic signals and applications tointerference-free spectrum, f0, and aperiodicity estimation,” inProc. ICASSP, Las Vegas, U.S.A., Apr. 2008, pp. 3933–3936.

[22] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen,
and Y. Wu, “LibriTTS: A corpus derived from LibriSpeech fortext-to-speech,” in Proc. INTERSPEECH, Graz, Austlia, Sep.
2019, pp. 1526–1530.

[4] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoderbased high-quality speech synthesis system for real-time applications,” IEICE Transactions on Information and Systems, vol. 99,
no. 7, pp. 1877–1884, Jul. 2016.

[23] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growingof GANs for improved quality, stability, and variation,” in Proc.
ICLR, Vancouver, Canada, May 2018.

[5] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever,and M. Welling, “Improved variational inference with inverse autoregressive flow,” in Proc. NIPS, vol. 29, Barcelona, Spain, Dec.
2016, pp. 4743–4751.

[24] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in Proc. CVPR, California, U.S.A., Jun. 2019, pp. 4401–4410.

[6] A. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals,
K. Kavukcuoglu et al., “Parallel WaveNet: Fast high-fidelity
speech synthesis,” in Proc. ICML, Stockholm, Sweden, Jul. 2018,
pp. 3918–3926.

[25] K. Oura, K. Nakamura, K. Hashimoto, Y. Nankaku, andK. Tokuda, “Deep neural network based real-time speech vocoderwith periodic and aperiodic inputs,” in Proc. SSW10, vol. 32, Vienna, Austria, Sep. 2019, pp. 13–18.

[7] W. Ping, K. Peng, and J. Chen, “Clarinet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR, New Orleans,
U.S.A., May 2019.

[26] Y.-C. Wu, T. Hayashi, T. Okamoto, H. Kawai, and T. Toda,“Quasi-periodic Parallel WaveGAN: A non-autoregressive raw
waveform generative model with pitch-dependent dilated convolution neural network,” IEEE/ACM Transactions on Audio,
Speech, and Language Processing, vol. 29, pp. 792–806, Jan.
2021.

[8] D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” in Proc. NeurIPS, vol. 31, Barcelona,
Spain, Dec. 2018, pp. 10 236–10 245.

[27] E. Song, R. Yamamoto, M.-J. Hwang, J.-S. Kim, O. Kwon, and J.M. Kim, “Improved Parallel WaveGAN vocoder with perceptuallyweighted spectrogram loss,” in Proc. SLT, online, Jan. 2021, pp.
470–476.

[9] R. Prenger, R. Valle, and B. Catanzaro, “WaveGlow: A flowbased generative network for speech synthesis,” in Proc. ICASSP,
Brighton, U.K., May 2019, pp. 3617–3621.
[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, vol. 27, Montreal, Canada, Dec. 2014,
pp. 2672–2680.

[28] R. Yamamoto, E. Song, M.-J. Hwang, and J.-M. Kim, “Parallel waveform synthesis based on generative adversarial networks
with voicing-aware conditional discriminators,” in Proc. ICASSP,
online, Jun. 2021, pp. 6039–6043.

[11] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh,
J. Sotelo, A. de Brébisson, Y. Bengio, and A. Courville, “MelGAN: Generative adversarial networks for conditional waveform
synthesis,” in Proc. NeurIPS, vol. 32, Vancouver, Canada, Dec.
2019, pp. 14 881–14 892.

[29] K. Mizuta, T. Koriyama, and H. Saruwatari, “Harmonic WaveGAN: GAN-based speech waveform generation model with harmonic structure discriminator,” in Proc. INTERSPEECH, online,
Sep. 2021, pp. 2192–2196.

[12] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A
fast waveform generation model based on generative adversarial
networks with multi-resolution spectrogram,” in Proc. ICASSP,
online, May 2020, pp. 6199–6203.

[30] M.-J. Hwang, R. Yamamoto, E. Song, and J.-M. Kim, “Highfidelity Parallel WaveGAN with multi-band harmonic-plus-noisemodel,” in Proc. INTERSPEECH, online, Sep. 2021, pp. 2227–
2231.

[13] J. Yang, J. Lee, Y. Kim, H. Cho, and I. Kim, “VocGAN: A highfidelity real-time vocoder with a hierarchically-nested adversarial
network,” in Proc. INTERSPEECH, online, Oct. 2020, pp. 200–
204.

[31] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional GANs,” in Proc. CVPR, Salt Lake City,
U.S.A., Jun. 2018, pp. 8798–8807.

[14] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial
networks for efficient and high fidelity speech synthesis,” in Proc.
NeurIPS, vol. 33, online, Dec. 2020, pp. 17 022–17 033.

[32] Y.-Y. Yang, M. Hira, Z. Ni, A. Chourdia, A. Astafurov, C. Chen,
C.-F. Yeh, C. Puhrsch, D. Pollack, D. Genzel et al., “TorchAudio:
Building blocks for audio and speech processing,” arXiv preprint
arXiv:2110.15018, Oct. 2021.

[15] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic
models,” in Proc. NeurIPS, vol. 33, online, Dec. 2020, pp. 6840–
6851.
[16] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan,“WaveGrad: Estimating gradients for waveform generation,” inProc. ICLR, online, May 2021.
[17] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “DiffWave: A versatile diffusion model for audio synthesis,” in Proc.
ICLR, online, May 2021.
[18] X. Wang, S. Takaki, and J. Yamagishi, “Neural source-filter
waveform models for statistical parametric speech synthesis,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 402–415, Nov. 2019.

802