StarGAN-VC2:

Rethinking Conditional Methods for StarGAN-Based Voice Conversion
StarGAN-VC2samples

Takuhiro Kaneko Hirokazu Kameoka Kou Tanaka Nobukatsu Hojo
NTT Communication Science Laboratories, NTT Corporation, JapanIntroduction
Limitations of one-to-one VC

Non-parallel multi-domain VC- Our goal is to learn mappings among
multiple domains (e.g., multiple speakers)
without relying on parallel data.

E.g., CycleGAN-VC [Kaneko+2017]- Achieves one-to-one VC in a
non-parallel setting.
- However, requires many generators toachieve multi-domain VC.

Speaker B(Their number increases according to the number of domains)

Speaker C

Adversarialloss

Adversarialloss

DY (ŷ)DX (x̂)GXY (x)GY X (ŷ)

Cycleconsistency loss

Cycleconsistency loss

Speaker A

Possible solution

Forward-inverse mapping

Speaker D

GY X (y)GXY (x̂)Inverse-forward mapping

StarGAN-VC [Kameoka+2018]- Extends CycleGAN-VC to a conditionalsetting by incorporating domain codes.
- Only requires a single generator.
- However, the quality is still low.
→ Challenge to address
Conditional
adversarial lossD(ŷ, c )G(x, c )Cycleconsistency loss

Objectivecc
cc

C(c |ŷ)Classiﬁcationloss

G(ŷ, c)c: Source domain code, c’ : Target domain code

Proposed method: StarGAN-VC2- Key ideas: We rethink conditional methods of StarGAN-VC in two aspects: training objectives and network architectures.

1. Rethinking conditional methods in training objectives

2. Rethinking conditional methods in G networks

i. (Previous) Classiﬁcation loss
- C is learned using real data.
- G tries to generate classiﬁable (i.e.,
far from the decision boundary) data.

i. Motivation- Accurate modulation translation is important to achieve
high-quality VC (e.g., GV [Toda+2007] & MS [Takamichi+2014] postﬁlters).

RealB

C(c |ŷ)ii. (Previous) Target conditional
adversarial loss
- D needs to simultaneously handle
hard negative (e.g., A→A) and easy
negative (e.g., B→A) samples.

Fake
A→A
Fake
B→ARealA
Classiﬁable

RealA

Fake
A→BRealB

FakeA

D(ŷ, c )Fake
B→AEasy negativeRealA

iii. (Proposed) Source and target
Fake
conditional adversarial lossB→A- This loss brings all the converted
data close to the target data in bothReal
source-wise and target-wise manners. Aii. (Previous) Channel-wiseDomain- Concatenated domain codes
code
are additively used.
- They cannot be directly used
Expandfor modulating data.

D(ŷ, c, c )

Fake
A→A

Fake
C→AConcat
Modulate

Fake
A→AReal Hard negativeAConv

γc' βc'iii. (Proposed) Modulation-based
Select- Domain codes are used to
select modulation parameters.
Domain code- They can be directly used forf − µ(f )CIN(f ; c ) = γc
+ β c
modulating data.
σ(f )Conditional instance normalization[Dumoulin+2017]

Experiments
i. Data- Dataset: Voice Conversion Challenge 2018
- Speakers: 4 Professional US English speakers(VCC2SF1, VCC2SF2, VCC2SM1, and VCC2SM2)- Sentences: 81 sentences (about 5 min.)- Sampling Rate: 22.05 kHz- Features: 34 MCEPs, log F0, APs (WORLD, 5 ms)

Objective evaluation
i. Evaluation metrics- Mel-cepstral distortion (MCD):
Global structural diﬀerence (smaller is better)- Modulation spectra distance (MSD):
Local structural diﬀerence (smaller is better)ii. Comparison of training objectives

Subjective evaluation
StarGAN-VC [Kameoka+2018] vs. StarGAN-VC2i. MOS for naturalnessAllii. Conversion process (Follow VCC 2018 baseline)- MCEP: StarGAN-VC2- log F0: Linear transformation- AP: No conversion→ WORLD vocoder [Morise+2016]

iii. Implementation and training- Network architectures are based on
CycleGAN-VC2 [Kaneko+2019](G: 2-1-2D CNN, D: 2D CNN).
- In training, no extra data, modules, or timealignment procedure are used.
- 4 × 3 = 12 diﬀerent source-and-target
mappings are learned in a single generator.

Intra-gender

StarGAN-VC

StarGAN-VC2- Improves both MCD and MSD.
Note: We ﬁx the conditional method in G networks
as modulation-based.

iii. Comparison of G networks

Inter-gender

MOS

Experimental conditions- StarGAN-VC2 outperforms StarGAN-VC
for every category.

ii. Preference score on speaker similarity
All
Intra-genderInter-gender- Improves MSD.
Note: We ﬁx the conditional method in the training
objectives as Lst-adv.

StarGAN-VC2Fair

StarGAN-VC- StarGAN-VC2 outperforms StarGAN-VC
for every category.
Copyright © 2019 Nippon Telegraph and Telephone Corporation