TEXT-TO-SPEECH SYNTHESIS USING STFT SPECTRA BASED ON
LOW-/MULTI-RESOLUTION GENERATIVE ADVERSARIAL NETWORKS
Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari
The University of Tokyo, Japan

1. SYNOPSIS
In Text-To-Speech (TTS) synthesis w/ Short-Term Fourier Transform (STFT) spectra:•[1](1) Proposes Generative Adversarial Networks (GANs) -based training algorithm using low-/multi-resolution spectra.

•(2) Demonstrates that the proposed algorithm using GANs w/ low-resolution spectra improves synthetic speech quality.

•2. CONVENTIONAL ALGORITHMSDNN-based TTS w/ STFT spectra[2]










(1) Generate raw STFT amplitude spectra.
(2) Reconstruct phase spectra by using
Griffin and Lim’s method[3].
Pros. avoiding a vocoding process
Cons. over-smoothing of amplitude spectra







GAN-based TTS w/ vocoders[4]Generated
𝒚 speechparams.

Linguistic 𝒙feats.

𝐿D 𝒚, 𝒚 = −Amplitude spectra: high-dimensional features
including spectral & excitation characteristics.
[3]
GAN-based TTS : effective for spectralenvelopes (i.e., mel-cepstral coefficients).
Assumption:
low-res. spectra ≒ spectral envelopes

Natural𝑡 log 𝐷 𝒚𝑡

𝐿D,1 𝒚(loss for natural params.)−Averagepooling𝑡 log 1 − 𝐷 𝒚𝑡

Linguistic 𝒙feats. & Log F0

𝐿ADV 𝒚 = 𝐿D,1 𝒚Low
𝐿G

naturalMinimizing
approx. JS-divergence••[References]



𝒚L







Low
ADV-Low: minimizing 𝐿G𝒚, 𝒚 (proposed)
Multi
ADV-Multi: minimizing 𝐿G𝒚, 𝒚 (proposed)

Results:
Averagepooling𝒚, 𝒚 = 𝐿MSE 𝒚, 𝒚

𝐿ADV 𝒚1:L(1) ADV & ADV-Multi didn’t improve speech quality.
⇒ Because of difficulty in minimizing
distribution differences in original resolution(2) ADV-Low improved speech quality.
⇒ Effect of the spectral envelopes compensation𝒚, 𝒚 = 𝐿MSE 𝒚, 𝒚

Low-res.
discriminative models
𝐷L ⋅L 𝔼𝒚 𝐿MSE
L+ 𝜔D 𝔼𝐿𝒚ADVL 𝐿ADV𝒚

L 𝔼𝒚 𝐿MSE
L+ 𝜔D 𝔼𝐿𝒚ADV
L 𝐿ADV𝒚

naturalGAN-based TTS w/ multi-res. STFT spectra

Multi𝐿G

MSE: minimizing 𝐿MSE
ADV: minimizing 𝐿G 𝒚, 𝒚 [4][2]𝒚, 𝒚Approximately emulates
the filter bank extraction

𝐷 ⋅𝐿G 𝒚, 𝒚 = 𝐿MSE 𝒚, 𝒚

Naturalspectral 𝒚
amplitudes

444 (linguistic feats, durations, Log F0, UV)/513/{74, 34, 14}(dim. of 𝒚(L) was changed in accordance with 𝑤)
Feed-Forward (see our paper)

Subjective evaluation (preference AB tests)

Generated𝒚 spectral
amplitudes 𝐿MSE 𝒚, 𝒚

Acousticmodels
𝐺 ⋅𝐿MSE 𝒚, 𝒚 = 𝒚 − 𝒚 ⊤ 𝒚 − 𝒚

𝔼𝒚 𝐿MSE+ 𝜔D 𝔼 𝐿
𝐿ADV 𝒚𝒚 ADV

DNN architectures

Minimizing
distribution difference𝒚1:

Dims. of 𝒙/𝒚/𝒚 L

GAN-based TTS w/ low-res. STFT spectra𝐿D,0 𝒚(loss for generated params.)

𝐺 ⋅

1.0

STFT analysis

Averagepooling1:
natural0: generated𝒚𝜔D and 𝜔DLDatasetGenerated(2) Update acoustic models 𝐺 ⋅ .
𝒙Average pooling

4,007 utterances taken fromJapanese female speaker(subset of JSUT[5] corpus, 3,808/199 for training/evaluation)
Frame length: 400, shift length: 80, FFT length: 1024,analysis window: Hamming
Zero padding: 6, pooling width 𝑤: {14, 30, 70}, stride: 𝑤 2

Naturalspeech 𝒚params.

Discriminativemodels
𝐷 ⋅Experimental conditions•

𝐿D 𝒚, 𝒚Acousticmodels
𝐺 ⋅(1) Update discriminative models 𝐷 ⋅ .

4. EXPERIMENTAL EVALUATION

Motivatione.g., 513 dim.



3. PROPOSED ALGORITHMS

𝔼𝒚 𝐿MSE+ 𝜔D 𝔼 𝐿
𝐿ADV 𝒚𝒚 ADV

ADV-Low

NaturalMSE

ADV

ADV-Low ADV-Multi(proposed) (proposed)(𝑤 = 14)MSE

ADV-Low(𝑤 = 30)

ADV-Low(𝑤 = 70): 𝑝 < 0.001: 𝑝 > 0.1

Examples of 513-dimensional amplitude spectra[1] Goodfellow et al., Proc. NIPS, 2014. [2] Takaki et al., Proc. INTERSPEECH, 2017. [3] Griffin et al., IEEE Trans. on ASLP, 1984. [4] Saito et al., IEEE/ACM Trans. on ASLP, 2018. [5] Sonobe et al., arXiv:1711.00354, 2017.

ICASSP 2018 SP-P6.6©Yuki Saito, 18/04/2018