18

Changing your voice and speaking style- Voice and prosody conversion with sequence-to-sequence model -Abstract
We propose an voice and prosody conversion method for impersonating a desired speaker’s identity and hiding aspeaker’s identity. The conversion method consists of acoustic feature conversion and time-domain neural
postfilter. The acoustic feature conversion is based on a sequence-to-sequence learning with attention
mechanism, which makes it possible to capture the long-range temporal dependencies between source andtarget sequences. The later post filter employs a cyclic model based on adversarial networks, which requires noassumption for the speech waveform modeling. In contrast to current voice conversion techniques, the proposed
method makes it possible to convert not only voice timbre but also prosody and rhythm while achieving highquality speech waveform generation due to the proposed time-domain neural post filter. The remaining challenge
is the real-time voice conversion which is our ongoing work.
Convert a speaker identity

Overview
Inputspeechbut, quit synthetic…

Convertedspeech

Acoustic feature conversion [1](Intonation, voice timbre, and rhythm)

Naturally-sounding!
OutputspeechTime-domain
neural post filter [2](e.g., Impersonating a speaker’s identity and modifying pronunciation)

Acoustic Feature Conversion [1]Speech samples→• Train encoders, decoder, and attention mechanism

Time warping function ofsource context information• Encoder-decoder model makes it possible toContext information
Extraction

1.Handle input and output sequencesof different lengths
2.Convert not only voice timbre
but also rhythm

Attentionmechanism

Acoustic feature prediction of the nexttime frame from the previous output
and the aligned context information

Source acoustic feature
Sourceencoder

Targetdecoder• Attention mechanism has the abilities to

1.Select critical information fromthe encoded representationin accordance with the output
sequence representation

Model training for minimizing
the differences toTarget acoustic feature
Targetdecoder

2.Consider the long-range temporaldependencies for converting intonation

Time-domain Neural Post Filter [2]Training
step

Minimize the prediction error

Conversion step

Use the predicted acoustic feature for predicting the next ones

Minimize the discrimination error

Train 𝐹 for imitating 𝐹(𝐲)1.Train the models with non-parallel
data of synthetic and natural speech

Real/Fake• Train conversion functions 𝐺, 𝐹 between synthetic and naturalspeechas naturalspeech by 𝐷• Cyclic model makes it possible toMinimize the reconstruction error

Train 𝐺, 𝐹 for minimizing
The differences to 𝐲ො

FakeDiscriminator

Minimize the discrimination error

Synthetic-to-natural

Train 𝐺 for imitating 𝐺 𝐱
as natural speech by 𝐷Real

Fake

Synthetic speech

Discriminator

Natural-tosynthetic

Train 𝐺, 𝐹 for minimizing
The differences to 𝐱ො
Minimize the reconstruction error

RealReal/Fake

2.Handle the phase information correctly
due to need for the reconstructionof speech waveform• Generative adversarial learning helps
to generate clear speech

Speech samples →(e.g., improving the speech quality)

Natural speech

References[1] K. Tanaka, H. Kameoka, T. Kaneko, N. Hojo, ” AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation
Mechanisms," in Proc. 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2019), May 2019.
[2] K. Tanaka, H. Kameoka, T. Kaneko, N. Hojo,“WaveCycleGAN2: Time-domain Neural Post-filter for Speech Waveform Generation," arXiv:1904.02892,
Apr. 2019, (submitted to INTERSPEECH2019.)

Contact
Kou Tanaka Email: cs-liaison-ml at hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.