ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 978-1-6654-0540-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICASSP43922.2022.9746713

ISTFTNET: FAST AND LIGHTWEIGHT MEL-SPECTROGRAM VOCODER
INCORPORATING INVERSE SHORT-TIME FOURIER TRANSFORM
Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, Shogo Seki
NTT Communication Science Laboratories, NTT Corporation, Japan
ABSTRACTIn recent text-to-speech synthesis and voice conversion systems, a mel-spectrogram is commonly applied as an intermediate representation, and the necessity for a mel-spectrogram
vocoder is increasing. A mel-spectrogram vocoder mustsolve three inverse problems: recovery of the original-scalemagnitude spectrogram, phase reconstruction, and frequencyto-time conversion. A typical convolutional mel-spectrogram
vocoder solves these problems jointly and implicitly using
a convolutional neural network, including temporal upsampling layers, when directly calculating a raw waveform. Suchan approach allows skipping redundant processes during
waveform synthesis (e.g., the direct reconstruction of highdimensional original-scale spectrograms). By contrast, theapproach solves all problems in a black box and cannot effectively employ the time-frequency structures existing in amel-spectrogram. We thus propose iSTFTNet, which replaces
some output-side layers of the mel-spectrogram vocoder with
the inverse short-time Fourier transform (iSTFT) after sufficiently reducing the frequency dimension using upsamplinglayers, reducing the computational cost from black-box modeling and avoiding redundant estimations of high-dimensional
spectrograms. During our experiments, we applied our ideas
to three HiFi-GAN variants and made the models faster andmore lightweight with a reasonable speech quality.1
Index Terms— Waveform synthesis, mel-spectrogram
vocoder, convolutional neural network, inverse short-time
Fourier transform, generative adversarial networks
1. INTRODUCTION
Speech is a frequently used modality in communication, andtext-to-speech (TTS) synthesis and voice conversion (VC)
have been studied to eliminate human-human and humanmachine boundaries. In both TTS and VC, typical methods
use a two-stage approach: (1) The first model predicts thetarget intermediate representation from the text or source intermediate representations. (2) The second step generates a
raw waveform from the predicted intermediate representation.
A mel-spectrogram is widely used as an intermediate representation in recent TTS [1, 2, 3, 4, 5] and VC [6, 7, 8] systems
owing to its compactness and expressiveness. Consequently,
the demand for a mel-spectrogram vocoder is increasing.
1 Audio samples are available at https://www.kecl.ntt.co.jp/
people/kaneko.takuhiro/projects/istftnet/.

978-1-6654-0540-9/22/$31.00 ©2022 IEEE

Mel-spectrogram (80 dim.)Mel-spectrogram (80 dim.)

Input conv

Input conv

ResBlock(upsample ×8)

ResBlock(upsample ×8)

F 1024
T1
F 128
T8

ResBlock(upsample ×8)

F 16
T 64

ResBlock(upsample ×8)

Replace

Output conv

ResBlock(upsample ×2)
ResBlock(upsample ×2)

Mag./Phase → iSTFT
Fast &
Raw waveform
Lightweight

Output conv
Raw waveform(a) Standard mel-spectrogram vocoder

T 256
F: FFT size
T: Time scale(F·T=const.)(b) iSTFTNet (ours)

Fig. 1. Comparison of a standard convolutional melspectrogram vocoder and iSTFTNet (ours). We propose replacing the output-side layers of the standard vocoder (a) with
iSTFT (b) when the number of frequency dimensions is sufficiently small (e.g., herein, the FFT size is 16) compared to
the number of dimensions of the input mel-spectrogram (80).
A mel-spectrogram vocoder must solve the following
three inverse problems: recovery of the original-scale magnitude spectrogram, phase reconstruction, and frequency-totime conversion. A typical convolutional mel-spectrogram
vocoder (e.g., a generative adversarial network (GAN [9])based model [10, 11, 12]) solves these problems jointly andimplicitly using a convolutional neural network (CNN), including temporal upsampling layers, when directly calculating a raw waveform from a mel-spectrogram. Such an approach allows omitting redundant processes during waveform
synthesis, e.g., the direct reconstruction of high-dimensional
original-scale spectrograms. However, this approach solves
all problems in a black box and cannot efficiently employ
time-frequency structures that exist in a mel-spectrogram.
We thus propose iSTFTNet, which replaces some outputside layers of the convolutional mel-spectrogram vocoder(Fig. 1(a)) with well-established signal processing, particularly an inverse short-time Fourier transform (iSTFT)(Fig. 1(b)) when the number of frequency dimensions (FFTsize of 16, Fig. 1) is sufficiently small compared to the numberof dimensions of the input mel-spectrogram (80, Fig. 1). This
reduces the computational cost from black-box modelingwhile avoiding redundant estimations of high-dimensional
original-scale spectrograms (FFT size of 1024, Fig. 1). During our experiments, we applied our ideas to three HiFiGAN variants [12] and made the models faster and more
lightweight with a reasonable speech quality.
The rest of this paper is organized as follows. In Section 2,
we discuss related studies. In Section 3, we review a typical

6207

ICASSP 2022

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:24:34 UTC from IEEE Xplore. Restrictions apply.

(2) Drop phase
Phase

2. RELATED WORK

(1) STFT

80 dim.

Raw waveform(3) Convert magnitude into mel-scale
Magnitude
Mel-spectrogram
513 dim.

Mel-spectrogram
extraction

convolutional mel-spectrogram vocoder and introduce iSTFTNet, which is a fast and lightweight variant. In Section 4, we
present the experiment results. In Section 5, we provide someconcluding remarks and areas of future research.

Mel-spectrogram vocoder (shortcut)

Neural vocoders have been studied in speech signal processing and machine learning. The first breakthrough wasachieved using autoregressive models, including WaveNet [13]and WaveRNN [14], which achieved an impressive quality but
slow inference speed owing to a sample-by-sample estimation. Parallelizable non-autoregressive models have therefore
gained attention. For example, Parallel WaveNet [15] andClariNet [16] distill an autoregressive teacher model into a
non-autoregressive convolutional student model. WaveGlow [17] eliminates the requirement for a teacher model byincorporating Glow [18], composed of affine coupling layers
and a 1 × 1 invertible convolution. WaveGrad [19] and DiffWave [20], based on diffusion probabilistic models [21, 22],
apply non-autoregressive CNNs for parallel computations.
A GAN [9]-based model [10, 11, 12, 23, 24, 25] achieves
parallelizable training and inference through noncausal convolutions. As described, CNNs with temporal upsamplinglayers, shown in Fig. 1(a), have been commonly used inrecent mel-spectrogram vocoders. Thus, beyond the HiFiGANs [12] used in our experiments, our ideas are generaland can be applied to other models.
The use of iSTFT for neural speech synthesis was previously introduced [26, 27, 28] (including our own early attempts [26]). As the main difference between the previous
models and iSTFTNet, the former requires a high-capacity
or high-computational model (e.g., 12 residual blocks with
2048 channels [28] and 2D CNNs [26, 27]) because they aim
to reconstruct the original-scale spectrograms without changing the time scale. By contrast, iSTFTNet employs a hybrid
approach in which iSTFT is applied after some upsampling
processes (Fig. 1(b)). This allows a reasonable performanceusing a low-capacity model (e.g., 1D CNNs, commonly usedin typical GAN vocoders [10, 11, 12, 23, 24, 25]).
3. METHOD
3.1. Convolutional mel-spectrogram vocoder
As shown in Fig. 2, the mel-spectrogram is extracted from theraw waveform as follows: (1) The magnitude and phase spectrograms are extracted from the raw waveform using a shorttime Fourier transform (STFT). (2) The phase spectrogram is
dropped. (3) The magnitude spectrogram is converted into a
mel-scale. Because a mel-spectrogram vocoder is aimed at aninverse process, three inverse problems must be solved: (3’)
recovery of the original-scale magnitude spectrogram; (2’)
phase reconstruction; and (1’) frequency-to-time conversion.
A typical convolutional mel-spectrogram vocoder solves
these problems jointly and implicitly using a CNN, including temporal upsampling layers, while directly calculating a
raw waveform from a mel-spectrogram. This approach al-

Fig. 2. Processing flows of mel-spectrogram extraction (lightblue) and mel-spectrogram vocoder (pink)lows redundant processes (e.g., the direct reconstruction ofhigh-dimensional original-scale magnitude and phase spectrograms) to be skipped during waveform synthesis. This allows a convolutional mel-spectrogram vocoder to solve theaforementioned problems with a low-capacity model. For example, HiFi-GAN V2 [12] achieves a good performance using only 1D convolutions of channels smaller than 128, despite being smaller than the original-scale spectrogram dimensions (i.e., 513, Fig. 2).
3.2. iSTFTNet: Fast and lightweight vocoder with iSTFTBlack-box modeling is useful for discovering potential shortcuts. However, we cannot effectively employ the timefrequency structures existing in the mel-spectrogram despite
providing a hint for solving inverse problems.
Thus, we propose iSTFTNet, which employs time and frequency structures explicitly using iSTFT after sufficiently reducing the frequency dimension using some upsampling layers, as shown in Fig. 3(b)–(d). Here, we utilize the characteristics of STFT, that is, the trade-off between the time and frequency resolution. More precisely, when the iSTFT required
after s× upsampling is represented as iSTFT(fs , hs , ws ),where fs , hs , and ws indicate the FFT size, hop length,
and window length, respectively; iSTFT(fs , hs , ws ) can becalculated using the parameters of iSTFT required for theoriginal-scale spectrogram, iSTFT(f1 , h1 , w1 ):
f1 h1 w1
, ,,(1)
iSTFT(fs , hs , ws ) = iSTFTs s swhere we utilize the aforementioned STFT characteristic, thatis, f1 · 1 = fs · s = constant. This equation means that we canreduce the frequency dimensions by increasing s. As shownin Fig. 3, we can simplify the structure in the frequency direction by increasing the number of upsamples. In Section 4.2,
we empirically found that simplification through more than
two upsamples (Fig. 3(c) or (d)) is essential for a faster andmore lightweight model to achieve a reasonable quality.
3.3. Implementation
iSTFTNets (Fig. 3(b)–(d)) have the mostly same network architecture as the baseline (Fig. 3(e)). Hence, when a reliableconvolutional mel-spectrogram vocoder is obtained, it is easy
to incorporate the concept of iSTFTNet. However, three minor but essential modifications are required: (i) The output
channels of the final convolutional layer should be changed
from 1 to (fs /2 + 1) × 2 to generate magnitude and phase

6208
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:24:34 UTC from IEEE Xplore. Restrictions apply.

ResBlock(upsample ×8)

Input conv
80

ResBlock(upsample ×8)

ResBlock(upsample ×2)

ResBlock(upsample ×2)

ResBlock
Output conv

Output conv
Phase

T
Mel-spectrogram

Output conv

513
Mag.
Mag.
T
iSTFT
(f1=1024, h1=256, w1=1024)
256T

5
Mag.

9
Mag.

65

Output conv
Phase

Phase

Phase

8T
iSTFT
(f8=128, h8=32, w8=128)
256T

64T
iSTFT
(f64=16, h64=4, w64=16)
256T

128T
iSTFT
(f128=8, h128=2, w128=8)
256T

Output conv
256T(e) C8C8C2C2 (standard)(d) C8C8C2I (ours)(c) C8C8I (ours)(b) C8I (ours)(a) C1I✓ Fast✓ Lightweight✓ High quality(Best balance model)

Fig. 3. Architectures of iSTFTNets (b)–(d) and a standard convolutional mel-spectrogram vocoder (e). The model is denoted
as Cx. . . (I), where Cx indicates the use of a residual block (ResBlock) [29] with an ×x upsampling layer and I indicates theuse of iSTFT. Here, the input 80-dimensional mel-spectrogram was extracted from a 22.05-kHz waveform using STFT with an
FFT size of 1024, hop length of 256, and window length of 1024.
spectrograms instead of a raw waveform. (ii) Exponential andsine activation functions should be applied to the output of(i) when calculating the magnitude and phase spectrograms,
respectively. (iii) A raw waveform should be generated fromthe magnitude and phase spectrograms using iSTFT (Eq. (1)).
For (ii), we use an exponential activation function because therequired magnitude spectrogram uses a linear scale, whereas
the input mel-spectrogram uses a log scale, and we apply asine activation function to represent the periodic characteristics of the phase spectrogram.
4. EXPERIMENTS
4.1. Experiment setup
Dataset. We used the LJSpeech dataset [30], consisting of
13,100 audio clips (24 h) of a female speaker. Here, 12,600,
250, and 250 utterances were used for the training, validation,and evaluation, respectively. The audio clips were sampled at
22.05 kHz, and 80-dimensional log-mel spectrograms wereextracted with an FFT size of 1024, hop length of 256, andwindow length of 1024.
Network architectures. We applied our ideas to three HiFiGAN variants [12] (high-quality (V1), light (V2), and carefully tuned (V3) variants). We implemented them based on an
open-source code2 for fair comparison with the various synthesis speeches provided. As mentioned in Section 3.3, withthe exception of the three modifications described above, weused the same architectures as the baselines.
Training settings. We trained the models using the HiFiGAN configuration provided in the open-source code2 , theparameters of which were tuned for stable training across various datasets. We trained the model for 2.5M iterations using the Adam optimizer [31] with an initial learning rate of
0.0002, and momentum terms β1 and β2 of 0.5 and 0.9, respectively. For the loss function, we used a combination ofleast squares GAN [32], mel-spectrogram [12], and feature
matching [33, 10] losses.
2 https://github.com/kan-bayashi/ParallelWaveGAN

4.2. Evaluation
We conducted a mean opinion score (MOS) test to evaluate the perceptual quality, randomly selecting 20 utterancesfrom the evaluation set and using the ground truth melspectrograms of the utterances as the vocoder input. This test
was conducted online with 16 listeners. Audio samples areavailable from the link1 presented on the first page. As an
objective metric, we used the conditional Fréchet wav2vec
distance (cFW2VD), which measures the distance between
real and generative distributions in a wav2vec 2.0 [34] feature
space conditioned on text information. This is conceptually similar to the Fréchet inception distance (FID) [35]and Fréchet DeepSpeech distance (FDSD) [36], which measure the perceptual quality of images and speeches, respectively. We used cFW2VD instead of conditional FDSD
(cFDSD) [36] to evaluate the raw waveform directly without converting it into a power spectrogram, as required in
cFDSD. We found that MOS has a higher correlation withcFW2VD than with cFDSD (Spearman’s rank correlation of-0.93 and -0.83, respectively). In cFW2VD, the smaller thevalue is, the better the perceptual quality. Table 1 shows theresults, along with the inference speed and model size. We
examined the approach from three perspectives.
(1) How many layers should be replaced with iSTFT? The
vocoder is improved by replacing its output-side layers witha faster and more lightweight iSTFT. To determine the number of layers to be replaced, we investigated the performancedifferences between the models shown in Fig. 3. The corresponding results are listed in Table 1 (Nos. 2–5, 7–10, and
12–14).3 As expected, the inference speeds up, and the model
size decreases with more replaced layers. For the MOS, wefound that C8I performs worse than the original in all cases;
however, C8C8I and C8C8C2I for V1 and V2 were comparable to the original. For V3 only, the performance decreases
when C8C8I is used. This is because V3 is carefully tuned
to reduce the number of layers and loses its generality. How3 C8C8C2I was not used for V3 because the network of V3 is C8C8C4
and does not have a fourth upsampling layer, differently from V1 and V2.

6209
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:24:34 UTC from IEEE Xplore. Restrictions apply.

Table 1. Comparison of MOS with 95% confidence intervals,
cFW2VD, inference speed, and model size. The inference
speed (relative speed compared to real time) using a GPU was
calculated on a single NVIDIA V100 GPU, and the speed using a CPU was computed on a MacBook Pro laptop (2.7-GHz
Intel Core i7). We report the average score over the utterancesin the evaluation set. The model identifier (e.g., C8C8I) is
shown in Fig. 3. The numbers in () indicate the rates (%)
compared with the baselines (V1, V2, or V3). The underlinedmodels are iSTFTNets (fast and lightweight models).
No.

Model

MOS↑ cFW2VD↓

Speed↑(GPU)

Speed↑(CPU)# Param↓(M)––

1 Ground truth

4.46 ±0.14

–

–

2
3
4
5
6

V1 (original) [12]V1-C8C8C2I
V1-C8C8I
V1-C8I
V1-C8C1I

4.22 ±0.174.22 ±0.174.26 ±0.173.32 ±0.22
3.82 ±0.17

0.020
0.018
0.020
0.073
0.033

×143.59 (100)×179.42 (125)×245.68 (171)×609.43 (424)×326.39 (227)

7 V2 (original) [12]8 V2-C8C8C2I
9 V2-C8C8I
10 V2-C8I
11 V2-C8C1I

3.91 ±0.17
3.98 ±0.17
3.95 ±0.16
3.21 ±0.20
3.44 ±0.20

0.046
0.038
0.042
0.096
0.071

×624.47 (100) ×10.39 (100)×732.96 (117) ×13.34 (128)×1025.46 (164) ×20.37 (196)×1720.91 (276) ×68.05 (655)×1081.37 (173) ×39.14 (377)0.93 (100)0.92 ( 99)
0.89 ( 96)
0.78 ( 84)1.30 (140)

12 V3 (original) [12]13 V3-C8C8I
14 V3-C8I
15 V3-C8C1I

3.78 ±0.16
3.41 ±0.19
2.89 ±0.172.82 ±0.21

0.052
0.055
0.156
0.116

×933.06 (100) ×10.40 (100)×1517.70 (163) ×21.48 (206)×2481.87 (266) ×66.83 (642)×1925.15 (206) ×41.16 (396)1.46 (100)1.42 ( 97)1.28 ( 87)
1.77 (121)16 MB-MelGAN [24] 3.54 ±0.21
17 PWG [11]3.47 ±0.21

0.078
0.066

×1070.95
×79.71

2.54
1.35

No.

Model

MOS↑

cFW2VD↓1

Ground truth

4.32 ±0.10–23Conformer-FS2 + V1
Conformer-FS2 + V1-C8C8I

4.09 ±0.124.25 ±0.11

0.216
0.214

4

Conformer-FS2 [38]3.66 ±0.15

0.242

iSTFT



f1 h1 w1
sb , sb , sb, where b is the number of sub-bands.

This approach remains for future research.6
4.3. Application to text-to-speech synthesis

×1.34 (100) 13.94 (100)×1.63 (122) 13.80 ( 99)
×2.33 (174) 13.26 ( 95)
×7.57 (565) 10.89 ( 78)
×3.97 (296) 19.15 (137)×17.95×0.70

Table 2. Comparison of MOS with 95% confidence intervals
and cFW2VD for TTS synthesis

ever, note that despite the performance decrease, V3-C8C8I
is still comparable with Parallel WaveGAN (PWG) [11] (Table 1 (No. 17)), while improving the inference speed.
(2) Necessity of combining upsampling and iSTFT. The
numbers of both the upsampling layers and residual blocks
differ between C8I and C8C8I, as shown in Fig. 3(b) and(c). To solve this problem, we examined the performance of
C8C1I4 which applies one upsampling but uses two residual
blocks, similar to C8C8I. The corresponding results are presented in Table 1 (Nos. 6, 11, and 15). We found that C8C1I
is still worse than C8C8I, indicating that reducing the frequency dimension using upsampling is essential for obtaining
a reasonable quality when applying iSTFT without significant
changes to the network architecture.5(3) Comparison with fastest baseline. One of the fastest
GAN vocoders is multi-band (MB) MelGAN [24], which increases the speed of MelGAN [10] by changing the synthesis
target from a full-band signal to lower-resolution sub-band
signals [37]. To examine the validity of this speed, we compared our models with MB-MelGAN. The corresponding
results are presented in Table 1 (No. 16). Here, V2-C8C8I
outperformed MB-MelGAN for MOS, reducing the model
size and achieving a comparable speed. Note that the multiband formulation and iSTFT are orthogonal and compatible,
and iSTFT can be incorporated into MB-MelGAN using
4 The model size of C8C1I is larger than that of C8C8I because in thelatter, the channels are halved in the second ResBlock after upsampling,whereas in the former, this is not conducted owing to the absence of upsampling.
5 We also examined non-upsampling models (particularly, C1I (Fig. 3(a))
and C1C1I). Finding that they suffer from training difficulties with a significantly lower speech quality, we omitted them from the experiments.

Next, we examine the effectiveness of our approach whenapplied to TTS synthesis, focusing on the difference inperformance between V1 and V1-C8C8I when combinedwith Conformer-FS2 [38] (a combination of Conformer [39]and FastSpeech 2 (FS2) [5]). Following [12], which shows
the utility of fine-tuning on HiFi-GAN, we fine-tuned the
combined models for 300k iterations in an end-to-end manner after training each model. We applied an open-sourcecode [40]7 for fair comparison with other speech samples
provided. We conducted a MOS test to evaluate the perceptual quality by randomly selecting 20 utterances from theevaluation set. This test was conducted online with 12 listeners. Audio samples are available from the link1 presented on
the first page.
Table 2 summarizes the results. We found that V1-C8C8I
not only achieves a comparable or better performance than V1
and Conformer-FS2, but also is comparable with the groundtruth. These results indicate that iSTFTNet does not compromise the speech quality, even for TTS synthesis.
5. CONCLUSION
To employ time-frequency structures in a mel-spectrogram
while avoiding redundant estimations of high-dimensional
original-scale spectrograms, we propose iSTFTNet, replacing
the output-side layers of a convolutional mel-spectrogram
vocoder with iSTFT after reducing the frequency dimension
using upsampling layers. The experiment results demonstratethat we can make the models faster and more lightweightusing iSTFT, and that upsampling processes are essential for
obtaining a reasonable quality. As discussed in Section 2,
CNNs with upsampling layers are used by various vocoders
beyond GAN-based versions. Hence, applying our idea tosuch vocoders will be interesting. We also concurrently investigate the utility of the inverse fast Fourier transform for arecurrent neural vocoder, and plan to examine the differencein performance in future studies to further validate the utilityof the inverse Fourier transform.
Acknowledgements. This work was partially supported by
JST CREST Grant Number JPMJCR19A3, Japan.
6 As another difference between iSTFTNet and MB-MelGAN, MBMelGAN requires an additional sub-band STFT loss for stable training,whereas iSTFTNet does not.
7 https://github.com/espnet/espnet

6210
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:24:34 UTC from IEEE Xplore. Restrictions apply.

6. REFERENCES[1] Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep
Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang,
Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, andYonghui Wu, “Natural TTS synthesis by conditioning WaveNet on mel
spectrogram predictions,” in Proc. ICASSP, 2018, pp. 4779–4783.
[2] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller, “Deep Voice 3:
Scaling text-to-speech with convolutional sequence learning,” in Proc.
ICLR, 2018.
[3] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu, “Neural speech synthesis with transformer network,” in Proc. AAAI, 2019,
pp. 6706–6713.
[4] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and
Tie-Yan Liu, “FastSpeech: Fast, robust and controllable text to speech,”in Proc. NeurIPS, 2019.
[5] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and
Tie-Yan Liu, “FastSpeech 2: Fast and high-quality end-to-end text tospeech,” in Proc. ICLR, 2021.
[6] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark
Hasegawa-Johnson, “Auto-VC: Zero-shot voice style transfer with only
autoencoder loss,” in Proc. ICML, 2019, pp. 5210–5219.
[7] Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu
Hojo, “CycleGAN-VC3: Examining and improving CycleGAN-VCs
for mel-spectrogram conversion,” in Proc. Interspeech, 2020, pp.
2017–2021.
[8] Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu
Hojo, “MaskCycleGAN-VC: Learning non-parallel voice conversion
with filling in frames,” in Proc. ICASSP, 2021, pp. 5919–5923.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David
Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,“Generative adversarial nets,” in Proc. NIPS, 2014, pp. 2672–2680.
[10] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin,
Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio,
and Aaron Courville, “MelGAN: Generative adversarial networks forconditional waveform synthesis,” in Proc. NeurIPS, 2019.
[11] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in Proc. ICASSP,
2020, pp. 6199–6203.
[12] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Proc. NeurIPS, 2020.
[13] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan,
Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv
preprint arXiv:1609.03499, 2016.
[14] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman
Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord,
Sander Dieleman, and Koray Kavukcuoglu, “Efficient neural audio
synthesis,” in Proc. ICML, 2018, pp. 2410–2419.
[15] Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan,
Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward
Lockhart, Luis Cobo, Florian Stimberg, Norman Casagrande, Dominik
Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner,
Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, andDemis Hassabis, “Parallel WaveNet: Fast high-fidelity speech synthesis,” in Proc. ICML, 2018, pp. 3918–3926.
[16] Wei Ping, Kainan Peng, and Jitong Chen, “ClariNet: Parallel wave
generation in end-to-end text-to-speech,” in Proc. ICLR, 2019.
[17] Ryan Prenger, Rafael Valle, and Bryan Catanzaro, “WaveGlow: A
flow-based generative network for speech synthesis,” in Proc. ICASSP,
2019, pp. 3617–3621.
[18] Diederik P. Kingma and Prafulla Dhariwal, “Glow: Generative flow
with invertible 1 × 1 convolutions,” in Proc. NeurIPS, 2018.
[19] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad
Norouzi, and William Chan, “WaveGrad: Estimating gradients forwaveform generation,” in Proc. ICLR, 2020.

[20] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro, “DiffWave: A versatile diffusion model for audio synthesis,” inProc. ICLR, 2021.
[21] Yang Song and Stefano Ermon, “Generative modeling by estimating
gradients of the data distribution,” in Proc. NeurIPS, 2019.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel, “Denoising diffusion probabilistic models,” in Proc. NeurIPS, 2020.
[23] Jinhyeok Yang, Junmo Lee, Youngik Kim, Hoonyoung Cho, andInjung Kim, “VocGAN: A high-fidelity real-time vocoder with ahierarchically-nested adversarial network,” in Proc. Interspeech, 2020,
pp. 200–204.
[24] Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie,“Multi-band MelGAN: Faster waveform generation for high-quality
text-to-speech,” in Proc. SLT, 2021, pp. 492–498.
[25] Ahmed Mustafa, Nicola Pia, and Guillaume Fuchs, “StyleMelGAN:
An efficient high-fidelity adversarial vocoder with temporal adaptive
normalization,” in Proc. ICASSP, 2021, pp. 6034–6038.
[26] Keisuke Oyamada, Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka,
Nobukatsu Hojo, and Hiroyasu Ando, “Generative adversarial networkbased approach to signal reconstruction from magnitude spectrogram,”in Proc. EUSIPCO, 2018, pp. 2514–2518.
[27] Paarth Neekhara, Chris Donahue, Miller Puckette, Shlomo Dubnov,
and Julian McAuley, “Expediting TTS synthesis with adversarial
vocoding,” in Proc. Interspeech, 2019, pp. 186–190.
[28] Alexey A. Gritsenko, Tim Salimans, Rianne van den Berg, Jasper
Snoek, and Nal Kalchbrenner, “A spectral energy distance for parallel speech synthesis,” in Proc. NeurIPS, 2020.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deepresidual learning for image recognition,” in Proc. CVPR, 2016, pp.
770–778.
[30] Keith Ito and Linda Johnson, “The LJ speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017.
[31] Diederik P. Kingma and Jimmy Ba, “Adam: A method for stochastic
optimization,” in Proc. ICLR, 2015.
[32] Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang,and Stephen Paul Smolley, “Least squares generative adversarial networks,” in Proc. ICCV, 2017, pp. 2794–2802.
[33] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo
Larochelle, and Ole Winther, “Autoencoding beyond pixels using alearned similarity metric,” in Proc. ICML, 2016, pp. 1558–1566.
[34] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael
Auli, “wav2vec 2.0: A framework for self-supervised learning ofspeech representations,” in Proc. NeurIPS, 2020.
[35] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard
Nessler, and Sepp Hochreiter, “GANs trained by a two time-scale update rule converge to a local Nash equilibrium,” in Proc. NIPS, 2017.
[36] Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark,
Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan,“High fidelity speech synthesis with adversarial networks,” in Proc.
ICLR, 2020.
[37] Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng
Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, Dan Su, and Dong Yu,“DurIAN: Duration informed attention network for multimodal synthesis,” in Proc. Interspeech, 2020, pp. 2027–2031.
[38] Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi,
Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li,
Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun
Wei, Wangyou Zhang, and Yuekai Zhang, “Recent developments on
ESPnet toolkit boosted by conformer,” in Proc. ICASSP, 2021, pp.
5874–5878.
[39] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang,
Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, andRuoming Pang, “Conformer: Convolution-augmented transformer forspeech recognition,” in Proc. Interspeech, 2020, pp. 5036–5040.
[40] Tomoki Hayashi, Ryuichi Yamamoto, Katsuki Inoue, Takenori
Yoshimura, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Yu Zhang,and Xu Tan, “ESPnet-TTS: Unified, reproducible, and integratableopen source end-to-end text-to-speech toolkit,” in Proc. ICASSP, 2020,
pp. 7654–7658.

6211
Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:24:34 UTC from IEEE Xplore. Restrictions apply.