INTERSPEECH 2023
20-24 August 2023, Dublin, IrelandiSTFTNet2:
Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNN
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Shogo Seki
NTT Communication Science Laboratories, NTT Corporation, Japantakuhiro.kaneko@ntt.com

Abstract

Mel-spectrogram

The inverse short-time Fourier transform network (iSTFTNet)has garnered attention owing to its fast, lightweight, and highfidelity speech synthesis. It obtains these characteristics using
a fast and lightweight 1D CNN as the backbone and replacingsome neural processes with iSTFT. Owing to the difficulty of
a 1D CNN to model high-dimensional spectrograms, the frequency dimension is reduced via temporal upsampling. However, this strategy compromises the potential to enhance thespeed. Therefore, we propose iSTFTNet2, an improved variant of iSTFTNet with a 1D-2D CNN that employs 1D and 2D
CNNs to model temporal and spectrogram structures, respectively. We designed a 2D CNN that performs frequency upsampling after conversion in a few-frequency space. This design facilitates the modeling of high-dimensional spectrograms
without compromising the speed. The results demonstrated that
iSTFTNet2 made iSTFTNet faster and more lightweight withcomparable speech quality.1
Index Terms: speech synthesis, neural vocoder, inverse shorttime Fourier transform, convolutional neural network, generative adversarial networks

Time: T
FFT: 1024

T×8
1D CNN
Temporalmodeling

Magnitude & phase spectrograms
T×8
1D CNN
Temporalmodeling

T×4
iSTFT

1D→2D

Waveform

Time: 64T
FFT: 16
Large temporal upsampling

T×64

Room for improvement(a) iSTFTNet (baseline)

Time: T
FFT: 1024

T×8
1D CNN
Temporalmodeling

1D→2D

T×1
2D CNN
Spectrogram
modeling

T×32
iSTFT

Waveform

Time: 8T
FFT: 128
Small temporal upsampling

T×8
Faster(b) iSTFTNet2 (proposed)

Figure 1: Comparison of iSTFTNet [20] and iSTFTNet2 (proposed). (a) Owing to the difficulty of a 1D CNN to model highdimensional spectrograms, it is necessary to reduce the frequency dimension sufficiently in iSTFTNet using large temporal
upsampling. (b) In contrast, iSTFTNet2 mitigates this difficultyby conducting 1D-to-2D conversion in an earlier stage and applying a 2D CNN that can capture local structures in spectrograms. This modification facilitates the reduction of the neuraltemporal upsampling by eight times (i.e., from ×64 to ×8) andenhances the inference speed.

1. Introduction

Among the GAN-based neural vocoders, one of the fastest
and most lightweight models is the inverse short-time Fouriertransform network (iSTFTNet) [20], which achieves fast,
lightweight, and high-fidelity speech synthesis using a fast andlightweight 1D CNN (e.g., HiFi-GAN [14]) as the backbone
and replacing some output-side neural processes with fast andlightweight inverse short-time Fourier transform (iSTFT). Particularly, iSTFTNet applies iSTFT after sufficiently reducing
the frequency dimension using large temporal upsampling (Figure 1(a)) to avoid modeling high-dimensional spectrograms,
which are difficult for a 1D CNN to represent. This technique
is essential for making the model faster and more lightweight
while maintaining speech quality; however, it compromises thepotential to improve the inference speed by applying iSTFT
with fewer temporal upsampling.
One possible solution is to conduct spectrogram conversion using a fully 2D CNN (e.g., [25–27]). However, the direct application of a 2D CNN requires a significant increase in
the calculation cost because it increases linearly according to
the frequency dimension (e.g., 80 in a mel-spectrogram). Alternatively, inspired by the success of combining 1D and 2D
CNNs [28], we propose iSTFTNet2, an improved variant of
iSTFTNet with a 1D-2D CNN, in which 1D and 2D CNNs areused to model the global temporal and local spectrogram structures, respectively (Figure 1(b)). Particularly, we designed a
2D CNN that conducts frequency upsampling after performing

Text-to-speech (TTS) synthesis and voice conversion (VC) havebeen extensively studied to obtain the desired speech. The twostage approach widely used in TTS and VC is as follows. The
first model predicts the intermediate representation (e.g., melspectrogram) from the input data (e.g., text or speech), whereas
the second model synthesizes speech from the predicted intermediate representation. This study focuses on the second
model, the neural vocoder, and attempts to make it faster andmore lightweight to broaden its applicability.
Various neural vocoders have been developed with advances in deep generative models. The pioneer is an autoregressive model (e.g., WaveNet [1] and WaveRNN [2]) that achieves
high-fidelity speech synthesis but suffers from slow inference
owing to sample-by-sample processing. Various parallelizablenon-autoregressive models have been developed to boost theinference speed. For example, successful models include a
distillation-based (e.g., Parallel WaveNet [3] and ClariNet [4]),flow (e.g., Glow [5])-based (e.g., WaveGlow [6]), diffusion
probabilistic model [7, 8]-based (e.g., WaveGrad [9] and DiffWave [10]), and generative adversarial network (GAN) [11]based (e.g., [12–24]) models. Among them, this study focuses
on a GAN-based model while prioritizing the flexibility of thearchitectural design and the ability of fast inference.
1 Audio samples are available at https://www.kecl.ntt.co.
jp/people/kaneko.takuhiro/projects/istftnet2/.

4369

10.21437/Interspeech.2023-1726

Mel-spectrogram

Temporal modeling

Magnitude & phase spectrograms

Input
1D Conv

T×8
1D ConvT

1D ResBlocks

T×8
1D ConvT

1D ResBlocks

Output1D Conv

T
–
C
7

8T
–
C/2
16

8T
–
C/2
3, 7, 11

64T
–
C/4
16

64T
–
C/4
3, 7, 11

64T
–
2F1
7

T
Time:
Frequency: –80
Channel:Kernel size: –

T×4
iSTFT

1D→2D
64T
F1
2
–

Waveform
256T
–
–
–

(a) iSTFTNet (baseline)
Mel-spectrogram

T
Time:
Frequency: –80
Channel:Kernel size: –Temporal modelingInput
1D Conv

T×8
1D ConvT

1D ResBlocks

T
–
C
7

8T
–
C/2
16

8T
–
C/2
3, 7, 11Spectrogram modeling

1D→2D

Magnitude & phase spectrograms

Input
2D Conv

2D Blocks

F×2
2D ConvT

8T
F2/8
C/4
3×3

8T
F2/8
C/4
3×3

8T
F2/4
C/8
3×4

T×32
iSTFT
8T 8T
F2/2 F2
C/16 2
3×4 3×4

8T
F2
2–

Waveform
256T
–
–
–

(b) iSTFTNet2 (proposed)

Figure 2: Overall architectures of iSTFTNet [20] and iSTFTNet2 (proposed) when incorporated into HiFi-GAN [14]. T and C
denote the temporal length and number of output channels, respectively. F1 indicates the frequency dimension of the spectrogram in
iSTFTNet, whereas F2 indicates that in iSTFTNet2. C is set to 128 when HiFi-GAN V2 (lightweight variant) is used as the backbone.
F1 and F2 can be calculated by f2s + 1 (see Equation 1 for the definition of fs ). When the FFT size is 1024, they are calculated as
F1 = 1024/64+ 1 = 9 and F1 = 1024/8+ 1 = 65.
2
2
temporal upsampling. This equation is based on the time andfrequency tradeoff, that is, f1 · 1 = fs · s = constant andindicates that the frequency dimension can be reduced s times
by conducting ×s temporal upsampling.
Figure 2(a) shows the overall architecture of iSTFTNet. We
present the architecture of iSTFTNet-C8C8I4,2 which is thebest balanced model that improves the speed and model size
while maintaining speech quality, where Cx indicates the useof 1D CNN blocks with ×x temporal upsampling, and Iy indicates the use of iSTFT with ×y temporal upsampling. Whenprioritizing the speed, a model that performs temporal upsampling fewer times, for example, iSTFTNet-C8C1I32, which
conducts temporal upsampling once, is better. However, it isshown that such a model deteriorates speech quality because of
the difficulty of a 1D CNN in modeling high-dimensional spectrograms [20].

sufficient conversion in a few-frequency space using 1D andfew-frequency 2D CNNs. This design facilitates the modelingof high-dimensional spectrograms, which are difficult for a conventional 1D CNN-based iSTFTNet to model, without compromising the speed. Furthermore, we propose an efficient module inspired by ShuffleNets [29, 30] to improve the speed andweight of the model further.
In the experiments, we examined the effectiveness of
iSTFTNet2 on two representative datasets: LJSpeech [31] (single English speaker) and VCTK [32] (multiple English speakers). The experimental results demonstrated that iSTFTNet2
made iSTFTNet faster and more lightweight with comparable
speech quality. Furthermore, we demonstrated the versatility of our ideas by applying iSTFTNet2 to multi-band modeling [16, 33], another technique for improving the speed. The
results showed that this variant could further improve the speedwith comparable speech quality.
The remainder of this paper is organized as follows. Section 2 briefly reviews the conventional iSTFTNet. Section 3
presents details of the proposed iSTFTNet2. Section 4 presents
our experimental results. Finally, Section 5 concludes the study
and discusses future research.

3. Proposal: iSTFTNet2
Considering the abovementioned facts, we attempted to construct an improved variant of iSTFTNet that can maintainspeech quality even with fewer temporal upsampling. A possible solution is to convert a spectrogram using a fully 2D
CNN that can capture the local structures in spectrograms(e.g., [25–27]). However, this replacement requires a significant increase in the calculation cost because it increases linearly in proportion to the frequency dimension (e.g., 80 in amel-spectrogram).
Alternatively, we developed iSTFTNet2, which constitutes
a 1D-2D CNN. Figure 2(b) presents the overall architecture of
iSTFTNet2. As shown in this figure, to model temporal structures efficiently, iSTFTNet2 uses the same 1D CNN for the first
three modules as that used in iSTFTNet, except that channel
concatenation is used instead of addition when integrating theoutputs of the multi-receptive fusion [14] in the 1D ResBlockto propagate more information to the subsequent 2D CNN. Unlike iSTFTNet, iSTFTNet2 conducts 1D-to-2D conversion in an
earlier step and applies a 2D CNN to effectively capture the lo-

2. Preliminary: Conventional iSTFTNet
iSTFTNet [20] is one of the fastest and most lightweight available neural vocoders. These characteristics were obtained by replacing some of the output-side layers of a fully neural vocoderwith fast and lightweight iSTFT. Particularly, it uses a fast andlightweight 1D CNN (e.g., HiFi-GAN [14]) as the backbone
with a high processing speed. However, it is challenging fora 1D CNN to model high-dimensional spectrograms becauseof the difficulty in capturing local structures in the frequency
direction. Hence, iSTFTNet reduces the frequency dimension
using temporal upsampling as follows:
f 1 h1 w 1
, ,
,
(1)
iSTFT(fs , hs , ws ) = iSTFTs s swhere fs , hs , and ws indicate the FFT size, hop length, andwindow length, respectively, required for the iSTFT after ×s

2 This is the same as iSTFTNet-C8C8 described in [20]. Here, we
added I4 to specify the temporal upsampling scale in the iSTFT.

4370

(a) 2D ResBlocks

×3

1,080 audio clips were used for training, validation, and evaluation, respectively. Following a study on HiFi-GAN [14], audio
clips were sampled at 22.05 kHz, and 80-dimensional log-mel
spectrograms were extracted from the audio clips with an FFT
size of 1024, a hop length of 256, and a window length of 1024.

Channel Concat

LReLU

3×3 Conv C/2

LReLU

3×3 Conv 2C

Channel Split

Channel Shuffle

3×3 Conv C

LReLU

3×3 Conv 2C

LReLU

×3

Implementation. We used iSTFTNet-C8C8I4 [20] as a baseline because it is the best-balanced model and its speech quality
has been demonstrated [20] to be comparable to that of HiFiGAN [14], which is a widely used baseline model. Specifically,
we implemented iSTFTNet-C8C8I4 based on HiFi-GAN V2 (alightweight variant) because we were interested in the performance of the lightweight model.3 We implemented iSTFTNet2
by replacing the modules of iSTFTNet-C8C8I4, as shown in
Figure 2. Particularly, we implemented the two variants using different 2D blocks, as shown in Figure 3. For clarity, we
denote iSTFTNet2 with 2D ResBlocks (Figure 3(a)) and thatwith 2D ShuffleBlocks (Figure 3(b)) as iSTFTNet2-Base and
iSTFTNet2-Small, respectively. As another comparison model,
we examined iSTFTNet-C8C1I32, which conducts the same
temporal upsampling as iSTFTNet2 but uses 1D ResBlocks instead of 2D blocks unlike iSTFTNet2. This model was used
to validate the importance of 2D blocks. All models were implemented based on open-source code,4 and the same training
settings were used. Specifically, a combination of least-squares
GAN [36], mel-spectrogram [14], and feature matching [12,37]
losses was used as the loss function. Each model was trained for
2.5M iterations using the Adam optimizer [38] with a batch sizeof 16, an initial learning rate of 0.0002, and momentum terms
β1 and β2 of 0.5 and 0.9, respectively.

(b) 2D ShuffleBlocks

Figure 3: Architectures of 2D blocks used in iSTFTNet2.
“LReLU” denotes a leaky rectified linear unit [34] with a negative slope of 0.1. “x × y Conv z” indicates a 2D convolution
with a kernel size of x × y and the number of output channels ofz, and C in this module denotes the number of input channelsof the block. In (a), residual connection (which is denoted by“+”) [35] is used. In (b), the weight-free operations used in
ShuffleNets [29, 30], i.e., “Channel Shuffle,” “Channel Split,”and “Channel Concat,” are used. In both blocks, the presented
block is stacked three times.
cal structures in the spectrograms, which are difficult for a 1D
CNN to model.
When considering the detailed configuration of a 2D CNN,
it is important to prevent an increase in the calculation cost
driven by the introduction of a 2D CNN because its calculation
cost increases linearly in proportion to the time and frequency
dimensions. To address this problem, iSTFTNet2 performs
the main conversion in a few-frequency space (specifically, 2D
blocks are applied in a space in which the frequency dimension
is downsampled eight times, as shown in Figure 2(b)), and thenconducts frequency upsampling in the last phase using transposed convolutions.
The design of the 2D blocks is a vital aspect to consider.
As shown in Figure 3, we developed two architectural designs.
The first is a 2D ResBlock (Figure 3(a)) that uses a residual connection [35] to propagate information efficiently. We adjusted
the model parameters (i.e., number of channels and kernel size)
such that the model became faster and more lightweight thaniSTFTNet-C8C8I4 (the best-balanced model). To further makethe model faster and more lightweight, we introduced a second
model, that is, a 2D ShuffleBlock (Figure 3(b)), which is inspired by efficient neural networks called ShuffleNets [29, 30].
In this block, the number of weight parameters used in the 2D
convolutional layers was adjusted such that it was half of thatof the 2D ResBlock (Figure 3(a)). Alternatively, in contrast to
the residual connection, the half channels are propagated directly without any addition to preserve the model capacity. A
channel shuffle [29, 30] is conducted to provide an interaction
between the skip and non-skip branches. Because the channel
shuffle, channel split, and channel concat are weight-free operations, this block is faster and more lightweight than the 2D
ResBlock. We demonstrated the empirical performance difference between the two 2D blocks in the experiments presented
in the next section.

Evaluation metrics. We conducted mean opinion score (MOS)
tests to evaluate perceptual quality. Twenty audio clips wererandomly selected from the evaluation set, and log-mel spectrograms extracted from the audio clips were used as vocoder
inputs. In addition to the speech synthesized by the abovementioned models, ground-truth speech was included as an anchor
sample. Ten listeners participated in each online test and wereasked to assess speech quality using a five-grade evaluation: 1= bad, 2 = poor, 3 = fair, 4 = good, and 5 = excellent. As an
objective metric, we used the conditional Fréchet wav2vec distance (cFW2VD) [20], which measures the distribution distance
between the real and synthesized speech in a wav2vec 2.0 [39]feature space conditioned to the text. The smaller the value, thebetter the speech quality. We evaluated the inference speed using a real-time factor (RTF) that was calculated by dividing theinference time by the duration of the synthesized speech (fixedat 1 s in the experiments). The RTF was measured using a single thread on an Intel Core i7-12700H CPU. The smaller thevalue, the higher the speed. We evaluated the model size using
the number of parameters (# Param). The smaller the value, themore lightweight the model. The audio samples are available
from the link on the first page.1
4.2. Results on single speaker dataset
Table 1 summarizes the results on LJSpeech. These results are
discussed from three perspectives.

4. Experiments
4.1. Experimental setup

Speech quality.
For the MOS test, we conducted theMann–Whitney U test. We found that iSTFTNet-C8C8I4,

Dataset. We examined the effectiveness of iSTFTNet2 using two representative datasets. LJSpeech [31], which includes
13,100 audio clips of a single English speaker, and 12,500, 100,and 500 audio clips were used for training, validation, and evaluation, respectively. VCTK [32], which comprises 44,081 audio
clips of 108 different English speakers, and 41,921, 1,080, and3 In additional experiments, we also examined the performance when
HiFi-GAN V1 (a high-quality variant) was used as a baseline, and observed a similar tendency in terms of cFW2VD, RTF, and # Param.
4 https://github.com/kan-bayashi/ParallelWaveGAN

4371

Table 1: Comparison of MOS with 95% confidence intervals,
cFW2VD, RTF, and # Param on LJSpeech. The numbers in ()indicate the rates (%) compared with HiFi-GAN V2.

Table 3: Comparison of MOS with 95% confidence intervals,
cFW2VD, RTF, and # Param when incorporating multi-bandmodeling on LJSpeech. The numbers in () indicate the rates(%) compared with HiFi-GAN V2.

Model

MOS↑

cFW2VD↓RTF↓

Ground truth

4.71 ±0.07––# Param↓
–

Model

MOS↑

cFW2VD↓RTF↓

HiFi-GAN V2

4.20 ±0.10

0.046

0.053 (100)0.93M (100)Ground truth

4.71 ±0.07

–

–

–

iSTFTNet-C8C8I4iSTFTNet-C8C1I32

4.12 ±0.10
3.71 ±0.13

0.042
0.071

0.029 ( 55)
0.018 ( 34)0.89M ( 96)
1.30M (140)iSTFTNet-MB

4.05 ±0.12

0.061

0.012 ( 22)0.82M ( 88)iSTFTNet2-MB

0.011 ( 21)0.83M ( 89)

4.24 ±0.10
4.22 ±0.10

0.036
0.040

0.021 ( 41)0.018 ( 35)0.85M ( 91)
0.79M ( 85)

4.25 ±0.110.040

iSTFTNet2-BaseiSTFTNet2-Small

tivated by this fact, we evaluated the performance of applying
iSTFTNet2 to multi-band modeling. We examined the effectiveness of this variant on LJSpeech [31].
Implementation.
As a baseline, we used iSTFTNetC4C4I4B4, where C/I/Bx indicates the use of 1D
blocks/iSTFT/multi-band modeling with ×x temporal
upsampling. We denote this model as iSTFTNet-MB. We
modified this model to iSTFTNet2 by replacing the second C4
with 2D ShuffleBlocks (Figure 3(b)) and using I16 instead of
I4. The number of final output channels of the 2D CNN (2in Figure 2) was modified to 8 to produce four sub-bands. To
allow for this expansion, we doubled the number of channels in
the 2D CNN and alternatively changed the number of output
channels in the first convolution layer in a 2D ShuffleBlock(Figure 3(b)) from 2C to C to make the model size andinference speed similar to those of iSTFTNet-MB. We denote
this model as iSTFTNet2-MB.
Results. Table 3 lists the results. The RTF and # Param were almost the same for iSTFTNet-MB and iSTFTNet2-MB because
we adjusted the model parameters of iSTFTNet2-MB such thatthey were almost the same. However, iSTFTNet2-MB significantly outperformed iSTFTNet-MB in terms of MOS (with a pvalue < 0.05 in the Mann–Whitney U test) and cFW2VD. Thisis possibly because it is easier to represent multiple sub-band
spectrograms simultaneously in a 2D CNN (in which channels
and frequencies are represented in independent dimensions)than in a 1D CNN (in which they are mixed in the same dimension). Furthermore, iSTFTNet2-MB was not significantly
different from HiFi-GAN V2 (Table 1) in these metrics (forthe MOS, the p-value > 0.05 in the Mann–Whitney U test),
while reducing the RTF to 21%. These results indicated that
iSTFTNet2-MB was the best among the variants of iSTFTNets
and iSTFTNet2s when prioritizing speed and speech quality.

Table 2: Comparison of MOS with 95% confidence intervals,
cFW2VD, RTF, and # Param on VCTK. The numbers in () indicate the rates (%) compared with HiFi-GAN V2.
Model

MOS↑

cFW2VD↓RTF↓

Ground truth

4.38 ±0.09

–

–

–

HiFi-GAN V2

3.99 ±0.11

0.061

0.053 (100)0.93M (100)iSTFTNet-C8C8I4iSTFTNet-C8C1I32

3.94 ±0.12
3.40 ±0.13

0.065
0.110

0.029 ( 55)
0.018 ( 34)0.89M ( 96)
1.30M (140)iSTFTNet2-BaseiSTFTNet2-Small

3.91 ±0.11
3.91 ±0.12

0.062
0.067

0.021 ( 41)0.018 ( 35)0.85M ( 91)
0.79M ( 85)# Param↓# Param↓iSTFTNet2-Base, and iSTFTNet2-Small were not significantly
different from HiFi-GAN V2 in terms of the p-values > 0.05.
In contrast, iSTFTNet-C8C8I32 performed significantly worse
than the others. cFW2VD was also the worst in iSTFTNetC8C8I32 and was comparable in the other cases. These results indicate that iSTFTNet2 can be used as an alternative to
iSTFTNet and HiFi-GAN regarding speech quality.
Inference speed. The RTF shows that both iSTFTNet2Base and iSTFTNet2-Small were faster than HiFi-GAN and
iSTFTNet-C8C8I4, achieving comparable speech quality.
iSTFTNet2-Small was the fastest among them and was comparable to iSTFTNet-C8C8I32, which sacrifices speech quality.
Model size. We found that iSTFTNet2-Base and iSTFTNet2Small were lighter than all baselines, and iSTFTNet2-Small was
the most lightest in terms of # Param.5
4.3. Results on multiple speaker dataset
Table 2 lists the results on VCTK. We observed that the sametendency as that observed on LJSpeech. For the MOS test,
iSTFTNet-C8C8I4, iSTFTNet2-Base, and iSTFTNet2-Small
were not significantly different from HiFi-GAN V2 in termsof p-values > 0.05 in the Mann–Whitney U test, whereas
iSTFTNet-C8C8I32 performed significantly worse than theothers. The RTF and # Param were the same as those observedon LJSpeech.

5. Conclusions
We proposed iSTFTNet2, an improved variant of iSTFTNet
that constitutes a 1D-2D CNN, in which 1D and 2D CNNs
are used to model temporal and spectrogram structures, respectively. The proposed architecture facilitated the application of
iSTFT to higher-dimensional spectrograms without large temporal upsampling, and the experimental results demonstrated
that iSTFTNet2 made iSTFTNet faster and more lightweight
while maintaining speech quality. Although we focused on
a GAN-based neural vocoder, our ideas have high applicability, and applying them to other models, including other neuralvocoders (e.g., [6, 9, 10]) and end-to-end text-to-speech synthesis (e.g., [4, 40–44]), remains the subject of future research.

4.4. Application to multi-band modeling
As discussed in [20], iSTFT and multi-band modeling [16, 33](another technique for improving speed) are complementary,and the speed can be further enhanced by combining them using
iSTFT fsb1 , hsb1 , wsb1 , where b is the number of sub-bands. Mo-

5 # Param of iSTFTNet-C8C1I32 is larger than that of iSTFTNetC8C8I4 because the number of channels is halved in the second
1D ResBlocks in iSTFTNet-C8C8I4, whereas it is not conducted in
iSTFTNet-C8C1I32 owing to the absence of temporal upsampling.
We used this strategy to confirm whether iSTFTNet-C8C1I32 could
not obtain comparable speech quality, even with expressive modules.

6. Acknowledgements
This work was supported by JST CREST Grant Number JPMJCR19A3, Japan.

4372

7. References[22] T. Kaneko, H. Kameoka, K. Tanaka, and S. Seki, “MISRNet:
Lightweight neural vocoder using multi-input single shared residual blocks,” in Proc. Interspeech, 2022, pp. 1631–1635.
[23] Y. Koizumi, K. Yatabe, H. Zen, and M. Bacchiani, “WaveFit: An
iterative and non-autoregressive neural vocoder based on fixedpoint iteration,” in Proc. SLT, 2022, pp. 884–891.
[24] T. Kaneko, H. Kameoka, K. Tanaka, and S. Seki, “Wave-U-Net
Discriminator: Fast and lightweight discriminator for generative
adversarial network-based speech synthesis,” in Proc. ICASSP,
2023.
[25] T. Kaneko, S. Takaki, H. Kameoka, and J. Yamagishi, “Generative
adversarial network-based postfilter for STFT spectrograms.” inProc. Interspeech, 2017, pp. 3389–3393.
[26] K. Oyamada, H. Kameoka, T. Kaneko, K. Tanaka, N. Hojo, and
H. Ando, “Generative adversarial network-based approach to signal reconstruction from magnitude spectrogram,” in Proc. EUSIPCO, 2018, pp. 2514–2518.
[27] P. Neekhara, C. Donahue, M. Puckette, S. Dubnov, andJ. McAuley, “Expediting TTS synthesis with adversarial vocoding,” in Proc. Interspeech, 2019, pp. 186–190.
[28] T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo, “CycleGANVC2: Improved CycleGAN-based non-parallel voice conversion,”in Proc. ICASSP, 2019, pp. 6820–6824.
[29] X. Zhang, X. Zhou, M. Lin, and J. Sun, “ShuffleNet: An extremely efficient convolutional neural network for mobile devices,” in Proc. CVPR, 2018, pp. 6848–6856.
[30] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “ShuffleNet V2: Practical guidelines for efficient CNN architecture design,” in Proc.
ECCV, 2018, pp. 116–131.
[31] K. Ito and L. Johnson, “The LJ speech dataset,” https://keithito.
com/LJ-Speech-Dataset/, 2017.
[32] J. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTKCorpus: English multi-speaker corpus for CSTR voice cloning
toolkit,” The Centre for Speech Technology Research, 2016.
[33] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo,
S. Kang, G. Lei, D. Su, and D. Yu, “DurIAN: Duration informed
attention network for multimodal synthesis,” in Proc. Interspeech,
2020, pp. 2027–2031.
[34] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities
improve neural network acoustic models,” in Proc. ICML, 2013.
[35] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in Proc. CVPR, 2016, pp. 770–778.
[36] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley,“Least squares generative adversarial networks,” in Proc. ICCV,2017, pp. 2794–2802.
[37] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther,“Autoencoding beyond pixels using a learned similarity metric,”in Proc. ICML, 2016, pp. 1558–1566.
[38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.
[39] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0:
A framework for self-supervised learning of speech representations,” in Proc. NeurIPS, 2020, pp. 12 449–12 460.
[40] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu,
“FastSpeech 2: Fast and high-quality end-to-end text to speech,”in Proc. ICLR, 2021.
[41] J. Donahue, S. Dieleman, M. Bińkowski, E. Elsen, and K. Simonyan, “End-to-end adversarial text-to-speech,” in Proc. ICLR,
2021.
[42] J. Kim, J. Kong, and J. Son, “Conditional variational autoencoderwith adversarial learning for end-to-end text-to-speech,” in Proc.
ICML, 2021, pp. 5530–5540.
[43] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, N. Dehak, and W. Chan, “WaveGrad 2: Iterative refinement for textto-speech synthesis,” in Proc. Interspeech, 2021, pp. 3765–3769.
[44] X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang,
Y. Leng, Y. Yi, L. He et al., “NaturalSpeech: End-to-end text
to speech synthesis with human-level quality,” arXiv preprint
arXiv:2205.04421, 2022.

[1] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals,
A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu,“WaveNet: A generative model for raw audio,” arXiv preprint
arXiv:1609.03499, 2016.
[2] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury,
N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord,
S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio
synthesis,” in Proc. ICML, 2018, pp. 2410–2419.
[3] A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals,
K. Kavukcuoglu, G. van den Driessche, E. Lockhart, L. Cobo,
F. Stimberg, N. Casagrande, D. Grewe, S. Noury, S. Dieleman,
E. Elsen, N. Kalchbrenner, H. Zen, A. Graves, H. King, T. Walters, D. Belov, and D. Hassabis, “Parallel WaveNet: Fast highfidelity speech synthesis,” in Proc. ICML, 2018, pp. 3918–3926.
[4] W. Ping, K. Peng, and J. Chen, “ClariNet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR, 2019.
[5] D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1 × 1 convolutions,” in Proc. NeurIPS, 2018, pp. 10 236–
10 245.
[6] R. Prenger, R. Valle, and B. Catanzaro, “WaveGlow: A flowbased generative network for speech synthesis,” in Proc. ICASSP,
2019, pp. 3617–3621.
[7] Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” in Proc. NeurIPS, 2019, pp. 11 918–
11 930.
[8] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic
models,” in Proc. NeurIPS, 2020, pp. 6840–6851.
[9] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan,“WaveGrad: Estimating gradients for waveform generation,” inProc. ICLR, 2021.
[10] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “DiffWave: A versatile diffusion model for audio synthesis,” in Proc.
ICLR, 2021.
[11] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, 2014, pp. 2672–2680.
[12] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh,
J. Sotelo, A. de Brébisson, Y. Bengio, and A. Courville, “MelGAN: Generative adversarial networks for conditional waveform
synthesis,” in Proc. NeurIPS, 2019, pp. 14 910–14 921.
[13] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A
fast waveform generation model based on generative adversarial
networks with multi-resolution spectrogram,” in Proc. ICASSP,
2020, pp. 6199–6203.
[14] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial
networks for efficient and high fidelity speech synthesis,” in Proc.
NeurIPS, 2020, pp. 17 022–17 033.
[15] J. Yang, J. Lee, Y. Kim, H. Cho, and I. Kim, “VocGAN: A highfidelity real-time vocoder with a hierarchically-nested adversarial
network,” in Proc. Interspeech, 2020, pp. 200–204.
[16] G. Yang, S. Yang, K. Liu, P. Fang, W. Chen, and L. Xie, “Multiband MelGAN: Faster waveform generation for high-quality textto-speech,” in Proc. SLT, 2021, pp. 492–498.
[17] A. Mustafa, N. Pia, and G. Fuchs, “StyleMelGAN: An efficient
high-fidelity adversarial vocoder with temporal adaptive normalization,” in Proc. ICASSP, 2021, pp. 6034–6038.
[18] J.-H. Kim, S.-H. Lee, J.-H. Lee, and S.-W. Lee, “Fre-GAN: Adversarial frequency-consistent audio synthesis,” in Proc. Interspeech, 2021, pp. 2197–2201.
[19] T. Okamoto, T. Toda, and H. Kawai, “Multi-stream HiFi-GAN
with data-driven waveform decomposition,” in Proc. ASRU, 2021,
pp. 610–617.
[20] T. Kaneko, K. Tanaka, H. Kameoka, and S. Seki, “iSTFTNet: Fastand lightweight mel-spectrogram vocoder incorporating inverse
short-time Fourier transform,” in Proc. ICASSP, 2022, pp. 6207–
6211.
[21] S.-H. Lee, J.-H. Kim, K.-E. Lee, and S.-W. Lee, “Fre-GAN 2:
Fast and efficient frequency-consistent audio synthesis,” in Proc.
ICASSP, 2022, pp. 6192–6196.

4373