GENERATIVE MOMENT MATCHING NETWORK-BASED RANDOM MODULATION POST-FILTERFOR DNN-BASED SINGING VOICE SYNTHESIS AND NEURAL DOUBLE-TRACKING†††‡†Hiroki Tamaru , Yuki Saito , Shinnosuke Takamichi , Tomoki Koriyama , and Hiroshi Saruwatari†The University of Tokyo, Japan

1. SYNOPSIS
We propose a generative moment matching network (GMMN)-based post-filter to(1) give random inter-utterance pitch variations to deterministically synthesized singing voices(2) give double-trackedness to singing voices••[1]DNN-based singing voice synthesisMusical score

Context

Speechparameters

DNN••

MSE

𝑡=2Target•𝑡=𝑇

Our post-filter
Generated
pitch contour 𝒚＊MSE: Mean squared error
•

Phase

Problem: Generated voice has no variation•𝒔𝒚 (1)

MS𝑺𝒚

Pitch (1 = semitone)

Segment-wise noise••𝒔𝒚 (𝑇′)𝒔𝒚 (𝑇′)MS𝑺𝒚•⋯
Minimizediscrepancy
of moments
Naturallyrandom
ISTFT
pitch contours⋯

Mix original voice with its copy modulated using ourpost-filter to produce double-trackedness
Spectrum, U / VOscillation

F0480 ms Hanning window, 240 ms segment shift1st order MS and 10 dim. uniform random noise
Subjective evaluation, 25 listeners, crowdsourcing

Example of generated pitch contours
Non-filtered
Post-filtered (4 lines)

ugigisasa

u
0.5

1.0

1.5

2.0

2.5

Score

DNN

F0
Post-filter

OriginalNDT

Perception rate of inter-utterance variation
ProposedControl(Two post-filtered versions) (The same version twice)
0.276
0.176

Modulated

p-Value
7.45 × 10-3

Our post-filter produces perceptible inter-utterance
variations!
•Double-trackedness score
Sample length

Problem: Unnatural sound artifacts[References]

127 dim. (40-dim. mel-cepstral coefficients, Log F0, etc.)

0.0

Neural double-tracking (NDT)

••

STFT

Feed-Forward (see our paper)Time [s]𝑈[−𝟏, 𝟏)Time [s]•⋯Mix original voice with its copy modulated using low
frequency oscillator to produce double-trackedness

•

MS𝑺𝒚

⋯

[2]
Artificial double-tracking (ADT)

Modulated
Modulated
Original
Original

100th iter.

Naturalpitch contour 𝒚STFT𝒔𝒚 (1)

10th iter.

(a) For synthesis model: 58 songs (two female singers)(b) For post-filter: 28 songs (one female singer)(c) For evaluations: 3 songs (one female singer)

DNN architecture
Speech params.
(including Δ & ΔΔ)
STFT settings for MS
Cond. GMMN input
Evaluation settings

Output
1st iter.

Randomness

Experimental conditions
Singing voice corpora

DNN that minimizes discrepancy of moments
between output and target ⇒ enables random
sampling from target distribution

Continuous F0
U/V

…

…

…

Musicalcontext•

Conditional GMMN[4]SpectrumOur approach

4. EXPERIMENTAL EVALUATION

Short-time Fourier transform (STFT) of parametersequence ⇒ represents envelope of sequence

Singingvoice

𝑡=1
Linguistic
context[3]
Modulation spectrum (MS)Current system

Randomness

3. GMMN-BASED POST-FILTER AND
NEURAL DOUBLE TRACKING

2. CONVENTIONAL METHODS•

HumanPitch(1 = semitone)•‡Tokyo Institute of Technology, Japan

Middle (approx. 4.88 s)
Long (approx. 10.24 s)

Proposed(NDT)
0.724
0.736

Conventional(ADT)
0.276
0.264

p-Value< 10-10< 10-10

Our method achieves more double-trackedness thanconventional method!

[1] Nishimura et al., Proc. INTERSPEECH, 2016. [2] Izhaki, Mixing Audio: Concepts, Practices, and Tools, 2017. [3] Takamichi et al., IEEE TRANS. AUDIO SPEECH LANG. PROCESS, 2016. [4] Ren et al., Proc. NIPS, 2016.

ICASSP 2019 SLP-P22.10©Hiroki Tamaru, 19/05/2019