NON-PARALLEL VOICE CONVERSION USING VARIATIONAL AUTOENCODERSCONDITIONED BY PHONETIC POSTERIORGRAMS AND D-VECTORS†‡‡‡†Yuki Saito , Yusuke Ijima , Kyosuke Nishida , and Shinnosuke Takamichi†The University of Tokyo, Japan••

1. SYNOPSIS
Our approaches for non-parallel Voice Conversion (VC) using Variational AutoEncoders (VAEs):(1) Introduce Phonetic PosteriorGrams (PPGs) for dealing with speech quality degradation.
(2) Extend conventional one-to-one VC to many-to-many VC (any speakers to any other speakers).
[1]
2. CONVENTIONAL VAE-BASED VC•3. PROPOSED VAE-BASED VC

Training VAEs for VC
Inputspeechparams. 𝒙
EncoderGeneratedspeech
𝒙 params.

Latentfactors
𝒛𝑞𝜙 𝒛|𝒙

VC using VAEs w/ PPGs and 𝑑-vectors

Decoder

𝑝𝜃 𝒙|𝒛, 𝒚s𝒚sSpeaker
codes[2]𝒩 𝒛; 𝟎, 𝑰ℒ 𝜽, 𝝓; 𝒙, 𝒚s = −𝐷KL 𝑞𝝓 𝒛|𝒙 || 𝒩 𝒛; 𝟎, 𝑰
Regularization term of 𝒛•𝒙

Conversion using trained VAEs𝒙

Encoder𝑞𝜙 𝒛|𝒙, 𝒛p𝒙𝒛

Encoder

Adaptation
dataDecoder•••••(1) Converted speech quality is degraded
due to vanishing phonetic properties.
(2) It supports only one-to-one VC.

••••••••

Speech params.
(including Δ & ΔΔ)
DNN architecture
Dim. ofPPGs/𝑑-vec./𝒛40-dim. mel-cepstral coefficients, Log F0, 10-dim. bap
Feed-Forward (see our paper)
56 (time variant)/16 (time invariant)/64
One-to-one: VC models are trained with corpora (b).
Many-to-many: VC models are trained with corpora (a).

VC settings•

Objective evaluation (Mel-Cepstral Distortion)
One-to-one(a) M2M(b) M2F𝒙Many-to-many(a) M2M(b) M2F

Worse9.0𝒛

Decoder𝑝𝜃 𝒙|𝒛, 𝒛p , 𝒛s

𝑑-vec.[4] Latent factors ofspeakers
𝒛s ⇒Pre-trainedspeakerverification

Assumption: 𝒛 is independent of 𝒚𝑠⇒ Expected to represent phonetic contents.

Problems

Speech corpora(a) For recognition/verification models: including 260 speakers(130 male and 130 female speakers)(b) For VC models: divided parallel 425 utterances(2 male and 1 female speakers)

8.0
7.0
6.0
5 1025 50 100 200 5 1025 50 100 200 5 10

Number of training data

(spkr. codes)

Subjective evaluation (MOS/DMOS tests)

Even unknown speaker can be⇒
involved in VC process!

Estimating unknown speaker representation(1) Speaker code adaptation[6]Using backprop. to adapt speaker codes(2) 𝑑-vector averaging
Using averaged 𝑑-vec. in voiced region of
the speaker’s adaptation data

(𝑑-vec.)(M2M: Male-to-Male, M2F: Male-to-Female)*DNN trained w/ parallel data•

25 50 100 200 5 10Better
25 50 100 200

Number of adaptation data
Proposed
Proposed

Conventional

Reference*𝒚s•

PPGs[3]
Latent factors of𝒛p ⇒phonetic contentsaPre-trained
ispeechu
recognition

Experimental conditions•(1) Phonetic contents are given as PPGs[4].
⇒ Phonetic contents can be restored!
(2) Discrete speaker codes are replaced with[5]continuous 𝑑-vectors .
⇒ Any speakers’ characteristics can beconverted into any others!

+ 𝔼𝑞𝝓 𝒛|𝒙 log 𝑝𝜽 𝒙|𝒛, 𝒚s
Reconstruction error of 𝒙

4. EXPERIMENTAL EVALUATION

MCD [dB]•‡NTT Corporation, Japan(a) Naturalness(b) Speaker similarity

ReferenceOne-tooneMany-tomany

Conventional
Proposed

Improved!

Improved!

M2M
M2F

M2MM2F(spkr. codes)

Proposed(𝑑-vec.)

Proposed(spkr. codes)

Proposed(𝑑-vec.)
1.0

2.0

3.0

MOS scores

4.0

5.0 1.0

2.0

3.0

4.0

DMOS scores

5.0[References][1] Hsu et al., Proc. APSIPA ASC, 2016. [2] Hojo et al., IEICE Trans. on Info. And Syst., 2017. [3] Dilokthanakul et al., arXiv:1611.02648, 2016. [4] Sun et al., Proc. ICME, 2016. [5] Variani et al., Proc. ICASSP, 2014. [6] Luong et al., Proc. ICASSP, 2017.

ICASSP 2018 SP-P6.1©Yuki Saito, 18/04/2018