HumanACGAN: conditional generative adversarial network with human-based auxiliary classifier
and its evaluation in phoneme perception
ICASSP 2021
paper ID: 1681

Yota Ueda1, Kazuki Fujii2, Yuki Saito1, Shinnosuke Takamichi1, Yukino Baba3 and Hiroshi Saruwatari1
1. The University of Tokyo, Japan, 2. National institute of Technology, Tokuyama College, Japan, 3. University of Tsukuba, Japan

1. SYNOPSIS
1. Proposes a generative adversarial network (GAN) that represents conditional human acceptable distributions.
2. Demonstrates that the proposed algorithm successfully trains a generator.

2. RESEARCH BACKGROUND

Real-data
distribution

Conditional human-acceptable
distribution for practical use(text-to-speech etc.)

Real-data
distribution

Humanacceptable
distribution

ConditionalGAN[1]

HumanGAN[2]

Unconditional

ACGAN
[3]

HumanACGAN(ours)

Discriminator 𝐷𝑆

Classifier 𝐷𝐶

DNN

HumansHumansHumanACGAN

Modeling method

Human-acceptable distribution

Generator 𝐺

Maximizing 𝐿𝑆 + λ𝐿𝐶 during generator training
𝐿𝑆 + λ𝐿𝐶 (𝐿𝐶 = 𝐷𝐶 𝑥𝑛 , 𝑐𝑛 )Estimating gradient using NES
𝜕(𝐿𝑆 +λ𝐿𝐶 )1≈𝜕𝑥𝑛
2𝜎 2 𝑅𝑅𝑟=1𝑟∆𝐷𝑆 𝑥𝑛3. RELATED WORKSClassifier 𝐷𝐶-ACGAN [3]

DNN

DNN

DNN

GAN [1]Minimizing 𝐿𝑆 = log(1 − 𝐷𝑆 𝑥𝑛 ) during generator training
Gradient is calculated analytically.
𝐷𝑆Real/Fake𝑥𝑛

HumanGAN [2]
Maximizing 𝐿𝑆 = 𝐷𝑆 𝑥𝑛 during generator training
Estimating gradient using Natural evolution strategy (NES) [4]Backpropagation using1𝜕𝐿𝑆 /𝜕𝑥𝑛 ≈ 2
2𝜎 𝑅𝑅
𝑟=1 ∆𝐷𝑆𝑟𝑥𝑛𝑟∙ ∆𝑥𝑛

ACGAN [3]Maximizing −𝐿𝑆 + λ𝐿𝐶 during generator training
𝐿𝐶 = log 𝐷𝐶 𝑥𝑛 , 𝑐𝑛 + log 𝐷𝐶 𝑥𝑛 , 𝑐𝑛 )Gradients are calculated analytically.
𝐷𝐶𝑟+ λ∆𝐷𝐶 𝑥𝑛 , 𝑐𝑛𝑟∙ ∆𝑥𝑛

Experimental conditions
CorpusAnalysis/synthesis
Optimization
Data space

JVPD [5]WORLD [6]Stochastic gradient descent
Applying PCA (principal component analysis) to log spectral envelope
and using 1st and 2nd PC as data space

Experiment 1: estimation of gradientExperiment 2: increase of objective function

MethodPreparing perturbated data 𝑥 + ∆𝑥 and 𝑥 − ∆𝑥
Synthesizing speech waveform from 𝑥 + ∆𝑥 and 𝑥 − ∆𝑥
Humans’ tasks
As discriminator: Evaluates difference in naturalness
acceptability 𝐷𝑆
As classifier: Evaluates difference in class acceptability 𝐷𝐶
Results
Gradients pointed to darker (i.e., higher posterior) zones.
Gradients for generator training were estimated correctly.

MethodTraining generator for 4 iterations using estimatedgradient
Perceptual test for generated data from initialized andtrained generator
Evaluated sample
Closed data (used in training) and open data
Results
Prior probabilities of naturalness and class acceptability
increased.
Generator was trained successfully.

Gradients bynaturalness acceptability

2nd PCGAN [1]
HumanGAN [2]Discriminator 𝐷𝑆DNN
HumansPrior

Backpropagation using

5. EXPERIMENTAL EVALUATIONSGenerator 𝐺DNNDNN

𝐺

Humans act as a discriminator and a classifier.

Gradients by classacceptability of /i/

16161212884400-4-4-8-8-12 -8 -4 0 4 8 12 -12 -8 -4 0 4 8 12
1st PC:Generated data of /i/

Gradients by classacceptability of /e/

1612840-4-8-12 -8 -4 0 4 8 12:Generated data of /e/

Posterior probability

Human-acceptable distribution
Humans can accept outside of real data

4. PROPOSED METHOD: HumanACGAN

Naturalness acceptability

high 1.0/i//e/

Class acceptability/i//e/: Closed data: Open data

0.80.6
0.40.2

low 0.0
Initialized Trained

References [1] Goodfellow et al., Proc. NIPS, 2014. [2] Fujii et al., Proc. ICASSP, 2020. [3] Mirza et al., arXiv:1411.1784, 2014. [4] Ilyas et al., Proc. ICML, 2018. [5] ” http:// research.nii.ac.jp/src/en/JVPD.html [6] Morise et al., IEICE, 2016.