HUMANGAN: GENERATIVE ADVERSARIAL NETWORK WITH HUMAN-BASEDDISCRIMINATOR AND ITS EVALUATION IN SPEECH PERCEPTION MODELING
Kazuki Fujii1,2 , Yuki Saito2 , Shinnosuke Takamichi2 , Yukino Baba3 , and Hiroshi Saruwatari2
12

National Institute of Technology, Tokuyama College, Japan.
Graduate School of Information Science and Technology, The University of Tokyo, Japan.
3
Faculty of Engineering, Information and Systems, University of Tsukuba, Japan.
ABSTRACT

We propose the HumanGAN, a generative adversarial network(GAN) incorporating human perception as a discriminator. A basic GAN trains a generator to represent a real-data distribution byfooling the discriminator that distinguishes real and generated data.
Therefore, the basic GAN cannot represent the outside of a real-data
distribution. In the case of speech perception, humans can recognize
not only human voices but also processed (i.e., a non-existent human) voices as human voice. Such a human-acceptable distribution
is typically wider than a real-data one and cannot be modeled by
the basic GAN. To model the human-acceptable distribution, we
formulate a backpropagation-based generator training algorithm byregarding human perception as a black-boxed discriminator. The
training efficiently iterates generator training by using a computerand discrimination by human. We evaluate our HumanGAN in
speech naturalness modeling and demonstrate that it can represent a human-acceptable distribution that is wider than a real-data
distribution.
Index Terms— generative adversarial network, human computation, black-box optimization, crowdsourcing, speech perception
1. INTRODUCTION
Generative models in machine learning have contributed strongly tomedia (e.g., speech)-related research. In particular, deep neural network (DNN)-based generative models, which are called “deep generative models,” can model a complicated data distribution thanks to
the nonlinear transformation of DNNs. A generative adversarial network (GAN) [1] is one of the most promising approaches in learning
deep generative models. The basic GAN framework includes not
only a generator but also a discriminator. The basic GAN trains thediscriminator by distinguishing real and generated data and separately trains the generator by fooling the discriminator. By iterating
the training steps, the generator can randomly generate samples that
follow a real-data distribution, i.e., training-data distribution. Many
studies on speech [2], image [3], and language [4] have successfully
applied the basic GAN framework, and more studies are expected toappear in speech-related research.
However, the basic GAN represents only a real-data distribution1 . Human speech perception can accept a deviation from realspeech. For instance, humans can recognize both human speech andprocessed (i.e., non-existent human) speech as human voice. In thispaper, we call such a human-acceptable distribution the perception
distribution and establish a GAN framework that can represent it.
1 This problem is common among generative models, but this paper deals
only with GAN.

GAN
Training
Generator Discriminator
Natural
Prior
Generateddistr.
data
Train to fool computer-based
discriminator.
Generation

Distr. of training data

Distr. of human perception

Fig. 1. Comparison of basic GAN and proposed HumanGAN.
Basic GAN trains generator by fooling DNN-based discriminator(i.e., computer-based discriminator), and generator finally represents
training-data distribution. In comparison, HumanGAN trains generator by fooling crowdworkers’ perceptual evaluation (i.e., humanbased discriminator), and generator finally represents humans’ perception distribution.
If the perception distribution covers wider ranges than a real-data
distribution, a basic GAN trained by fooling the discriminator could
never represent the perception distribution. Also, collecting a large
amount of real data for training the basic GAN would never solve
this problem.
In this paper, we propose the HumanGAN trained by foolinghuman perception to model a human perception distribution. The
HumanGAN regards a crowd as a black-boxed system that outputs a difference of posterior probabilities (i.e., to what degree dohumans accept the data) given generated data. We formulate abackpropagation-based generator training algorithm utilizing an optimization algorithm for the black-boxed system. Figure 1 shows acomparison of the basic GAN and proposed HumanGAN. Whereasa generator of the basic GAN fools a DNN-based discriminator, thatof the proposed HumanGAN fools perceptual evaluation performedby the crowdworkers of a crowdsourcing service. In this paper, weevaluate the HumanGAN in the modeling of speech naturalness,i.e., to what degree can humans accept synthesized speech as humanvoice. The experimental results demonstrate that 1) the perception
distribution covers a wider range than the real-data distribution,and 2) the proposed HumanGAN accurately models the perception
distribution, which the basic GAN cannot.
2. BASIC GAN USING DNN-BASED DISCRIMINATOR
The aim of the basic GAN is to match a generated-data distribution and real-data distribution. To achieve this, the basic GAN

Crowdworkers

based discriminator of the basic GAN with humans’ perceptual evaluation. The use of a DNN-based generator and prior noise are common between the basic GAN and HumanGAN. However, whereas
the basic GAN trains the generator by fooling the DNN-based discriminator, the HumanGAN trains one by fooling humans’ perceptual evaluation. Here, we re-define D (·) as a perception model that
represents humans’ perceptual evaluation. An input of the D (·) isx̂n generated from G (·), and the D (·) outputs a posterior probability about “to what degree is the input perceptually acceptable” from0 to 1. The objective function during training is as follows.

Perturbation
2

Worker’s answer to “to what degree
are two samples different?”
Backpropagation using
approximated[times]Fig. 2. Generator training procedure of proposed HumanGAN.
Crowdworkers state a perceptual difference (i.e., difference of posterior probabilities) of two perturbed samples. Answer and perturbation are used for backpropagation to train generator.
uses not only a generator G (·) that randomly generates data
but also a discriminator D (·) that distinguishes real and generated data. Here, let real data be x = [x1 , · · · , xn , · · · , xN ],and N be the number of data. The generator G (·) transforms
prior noise z = [z 1 , · · · , z n , · · · , z N ] into generated data x̂ =[x̂1 , · · · , x̂n , · · · , x̂N ]. The prior noise follows a known probability distribution, e.g., a uniform distribution. The input of thediscriminator D (·) is real data xn or generated data x̂n , and theD (·) outputs a posterior probability that the input is real data. An
objective function in training is formulated as
V (G, D) =N
X

log D (xn ) +NXlog (1 − D (G (z n ))) .

(1)

n=1

n=1

The generator G (·) and discriminator D (·) are trained one by one.
The following sections describe these training steps.
2.1. Generator training
The generator training step minimizes Eq. (1) to estimate the model
parameters of G (·). The model parameter θG is estimated:θG = argmin
θGNX

log (1 − D (G (z n ))) .

(2)

n=1

Namely, G (·) is trained to let D (·) recognize the generated data as“real.” In general, all computation processes are differentiable, andθG is iteratively estimated by using the backpropagation algorithm.
2.2. Discriminator training
The discriminator training step maximizes Eq. (1) to estimate themodel parameters of D (·). The model parameter θD is estimated:θD = argmax V (G, D) .
(3)
θD

2.3. Problem
When the generator is accurately trained, it can randomly sample
data that follows a real-data distribution. However, when the perception distribution covers ranges wider than the real-data distribution,
the basic GAN cannot represent the ranges. We show such an actual
example in speech perception in Section 4.2.
3. HUMANGAN USING HUMAN-BASED DISCRIMINATOR
We propose the HumanGAN to represent humans’ perception distribution. As shown in Fig. 1, the HumanGAN replaces the DNN-

V (G, D) =NX

D (G (z n )) .

(4)

n=1

As described in this equation, the HumanGAN never uses real-datain training.
3.1. Generator training
A model parameter θG of G (·) is estimated by maximizing Eq.
(4). In this paper, we formulate a gradient-based iterative method.
Namely, θG is iteratively updated as follows.
∂V (G, D)θG ← θG + α
,
(5)
∂θG
where α is the learning coefficient. The second term ∂V (G, D) /∂θG
is decomposed into a product of ∂V (G, D) /∂ x̂ and ∂ x̂/∂θG . In
the basic GAN, ∂V (G, D) /∂θG is estimated by using the standard backpropagation algorithm since the computation processesof both G (·) and D (·) are differentiable. However, D (·) is notdifferentiable in the HumanGAN. Therefore, we cannot estimate∂V (G, D) /∂ x̂ and cannot use the standard backpropagation algorithm. In this paper, we regard humans as a black-boxed systemthat outputs a difference of the posterior probabilities of generated
data and estimate ∂V (G, D) /∂ x̂ on the basis of an optimization
algorithm for black-boxed systems. We propose a training algorithm
that uses natural evolution strategies (NES) [5] that approximates
gradients by using data perturbations. Figure 2 shows the training
procedure.
(r)First, a small perturbation ∆xn is randomly
generated from a
multivariate Gaussian distribution N 0, σ 2 I . r is the perturbationindex (1 ≤ r ≤ R). σ is a constant value of standard deviation. I is
the identity
matrix. Then, a humano is presented with two perturbed
n
(r)
(r)
and evaluates the difference ofdata x̂n + ∆xn , x̂n − ∆xntheir posterior probabilities:




(r)
∆D(x̂(r)− D x̂n − ∆x(r). (6)n ) ≡ D x̂n + ∆xnn(r)∆D(x̂n ) ranges from −1 to 1. For example, we expect that a hu(r)man will return ∆D(x̂n ) = 1 when he/she perceives that D(x̂n +(r)(r)∆xn ) is greatly higher than D(x̂n −∆xn ). This perturbation andevaluation are iterated R times for one x̂n . Finally, these processes
are iterated for N generated data.
∂V (G, D) /∂ x̂ for the backpropagation algorithm is approximated as [5]∂V (G, D)∂V (G, D)∂V (G, D)∂V (G, D),··· ,,··· ,
,
=
∂ x̂
∂ x̂1
∂ x̂n
∂ x̂N
(7)
R
X

∂V (G, D)
1(r)=∆D(x̂(r)n ) · ∆xn .
∂ x̂n2σ 2 R r=1(8)

3.2. Discussion
Our work utilizes the idea of human computation [6] in which a machine performs its function by outsourcing inner steps to humans,
and the HumanGAN is regarded as a human-in-the-loop machinelearning technique. Some existing studies made humans have a function and integrated it into DNN training. Sakata et al. [7] regarded
humans as a feature extractor and improved the performance of amulti-class classifier. Also, Jaques et al. [8] regarded humans as an
auxiliary classifier and improved performances of a generator. Fromthese viewpoints, our work regards humans as a system that outputs
a difference of posterior probabilities and represents humans’ perceptual distribution.
Koyama et al.’s work [9] is related work from the viewpoint of
exploring humans’ perception. However, their aim is completely different from ours. Koyama et al. made humans score a perceptual
difference of two data and aimed at visualizing the perception distribution by interpolating the scores. In comparison, our work makes
humans score the perceptual difference of a perturbation and aims at
training a generator that represents the perception distribution.
Natural language-based image generation and editing [10, 11]
are related work from the viewpoint of modeling a data distributionthat corresponds to human intention. However, these methods only
estimate a conditional distribution within a real-data distribution. In
comparison, our work estimates a perception distribution even if it iswider than the real-data distribution.
The basic GAN utilizes only a computer-based discriminator,
and the HumanGAN utilizes only a human-based discriminator. One
part of our future work involves the use of a human-computer discriminator, i.e., the generator is trained by using both computerbased and human-based discriminators. Related studies have proposed active learning algorithms that use the basic GAN [12, 13].
These studies are currently limited to real-data distribution modeling but have the potential to be applied to perception distribution
modeling.
Data imbalance in training data is a general problem in machinelearning. Since the HumanGAN does not require real data for the
training, it has the possibility of relaxing the problem.
3.3. Practical limitation
Here, we list empirically found limitations of the HumanGAN.
As described in Eq. (4), the generator of the HumanGAN is
trained by maximizing the posterior probability. Therefore, a mode
collapse problem [14] is caused when data generated from the initial
generator is distributed near the mode of the perception distribution.
Also, a gradient vanishing problem is caused when generated data is
distributed far from the perception distribution. In the evaluation in
Section 4.3, we used real data for initializing the generator.
The setting of σ in the NES algorithm changes the degree of
perturbation. When σ is very small, the discriminator of the HumanGAN, i.e., crowdworkers, cannot find a perceptual difference, andit makes gradients vanish. When σ is very large, the gradient becomes inaccurate. In the preliminary experiments of the following
evaluation, we used several settings for σ and finally determined one
value.

ncipal component

Here, we summarize the number of queries to humans for training
the generator. Since the perturbation is performed R times for eachgenerated data, the total number of queries is a product of the number of training iterations, the number of generated data N , and thenumber of perturbations R.

08

6


0
-

 -6
 -8
2
-10

06

Real data

0
0
-

- 0 

6 8 1000

1st principal component

Fig. 3. Color map representing posterior probabilities in speech naturalness.
As described in Section 3.1, the number of queries to crowdworkers is a product of the number of training iterations, the number of data N , and the number of perturbations R. The number ofqueries increases the monetary and time costs for training. In thefollowing evaluation, we used a small generator, a small number ofgenerated data, and low-dimensional data to limit the costs.

4. EXPERIMENTAL EVALUATION
4.1. Experimental setup
An experimental evaluation was done to investigate whether our HumanGAN can represent the perception distribution of speech naturalness, i.e., the HumanGAN tries to represent a distribution of speechfeatures for which humans recognize the synthesized speech as human speech. Although the HumanGAN originally does not use realdata, this evaluation used speech data for comparison with the basic
GAN, dimensional reduction of speech features, and initialization of
the generator. The speech data used was reading-style speech uttered by 199 female speakers included in the Vowel Database: Five
Japanese Vowels of Males, Females, and Children Along with Relevant Physical Data (JVPD) [15]. Before extracting speech features,the speech data were downsampled at 16 kHz, and their powers werenormalized. The speech features were 513-dimensional log spectral envelopes extracted every 5 ms. The WORLD vocoder [16,17] was used for the feature extraction. Julius [18] was adopted
to obtain phoneme alignment, and we extracted speech features of
the Japanese vowel /a/. We applied principal component analysis(PCA) to the log spectral envelopes of /a/ of 199 female speakers
and extracted two-dimensional principal components. In this paper, we assumed that the real data of many speakers follows a twodimensional standard Gaussian distribution, and we normalized thetwo-dimensional principal components to have zero-mean and unitvariance. For humans’ perceptual evaluation, a speech waveform
was synthesized from the generated speech features. First, the firstand second principal components were generated from a generator
and de-normalized. The remaining speech features, i.e., the third andupper principal components, the fundamental frequency, and aperiodicity components, were copied from one frame of an averagevoice speaker. Average-voice speaker means a speaker whose firstand second principal components are the closest to 0. These correspond to speech features of one frame (5 ms). To make the perceptual evaluation easy for crowdworkers,we copied the features for
200 frames and synthesized 1 s of a speech waveform by using theWORLD vocoder.

Iameter   

1st update3  ate4  Fig. 4. Generated data (white point) at each training iteration. Color map shown in Fig. 3 was also drawn but never used in training.



Closed

O$%&


t#!
!b

or pro"b 
! 


 
P





1


Number of training iterations

Fig. 5. Posterior probability at each iteration step. Box indicatesfirst, second (i.e., median), and third quartiles. Line plot indicatesmean value.
4.2. Difference between real-data and perception distributionsFirst, to emphasize the need for the HumanGAN, we demonstratethat a perception distribution is different from a real-data distribution in speech naturalness. We split the two-dimensional space into
grids and evaluated humans’ naturalness tolerance (i.e., posterior
probability of speech naturalness). For the humans’ evaluation, we
instructed listeners to “listen to a voice and answer with 1 whenperceiving the speech as very unnatural and 5 when perceiving it
as very natural” and had listeners give a score on a 5-point scale(from 1 to 5). 1-through-5 of the obtained scores corresponded to
0.00, 0.25, 0.50, 0.75, 1.00 of the posterior probability D (x̂n ). The
evaluation was performed by using the Lancers crowdsourcing platform [19]. The posterior probability of each grid was set to the value
averaged among listeners. At least five listeners scored for one grid,and the total number of listeners was 96.
Figure 3 shows the result. As described in Section 4.1, the real
data was normalized to have zero-mean and unit-variance, and thebasic GAN represents this range. However, as shown in Fig. 3,
the perception distribution covered a range wider than the real-data
distribution. The basic GAN cannot represent this wider range, so
the HumanGAN, which adopts perceptual evaluation by humans isneeded for modeling the perception distribution.
4.3. Transition of generated data during training
Next, to intuitively understand generator training, we plotted the data
generated at each iteration step. The prior noise fed to the generator followed a two-dimensional uniform distribution U (−1, 1). Thisprior noise was randomly generated before a training iteration and
fixed during the iteration. The generator was a small feed-forward
neural network consisting of a two-unit input layer, 2 × 4-unit sigmoid hidden layers, and two-unit linear output layer. The model parameters were randomly initialized, but we iterated the random initialization until the initial generator output data that covered rangesof a higher posterior probability, shown in Fig. 3. The gradientdescent method with a learning rate of α = 0.0015 was used fortraining. Chainer [20] was used for implementation. The numberof generated data N , the number of perturbations, and the numberof training iterations were set to 100, 5, and 4, respectively. The
standard deviation of NES was set to 1.0. During the HumanGAN
training, we instructed listeners to “listen to two voices and answer
with 1 when perceiving the first one as significantly natural and 5
when perceiving the second as significantly natural. Answer 3 whenperceiving the two samples as equally natural.” The 1-through-5 ofthe obtained scores corresponded to 1.0, 0.5, 0.0, −0.5, −1.0 of the(r)difference of the posterior probabilities, ∆D(x̂n ).
Figure 4 shows the data generated for each iteration step. The
posterior probability of Fig. 3 is also drawn, but note that we neverused it in training. The figure indicates that the generated data transited from a lower probability range (darker area) to a higher probability range (brighter area) during the training iteration. Therefore,
we can qualitatively demonstrate that the HumanGAN makes it possible to train a generator that represents a human’s perception distribution.
4.4. Increase of posterior probability in training
Finally, we confirmed that the training iteration increases the posterior probability. Here, we sampled new prior noise not used intraining and generated new speech features. We call speech featuresgenerated from prior noise used in training “closed data” and thosegenerated from prior noise not used in training “open data.” Theposterior probabilities of the open data were scored by humans inthe same manner as in Section 4.2.
Figure 5 shows a box plot at each iteration step. We can see thatthe training iteration increases the posterior probabilities of not only
the closed data but also the open data. Therefore, we can quantitatively demonstrate that the generator of the HumanGAN can represent the perception distribution.
5. CONCLUSION
This paper presented the HumanGAN, which can represent humans’
perceptually recognizable distribution. The DNN-based discriminator of the basic GAN was replaced with human-based perceptual evaluation. The DNN-based generator of the HumanGAN wastrained by using a black-boxed optimization algorithm and humanevaluation. We evaluated the HumanGAN in speech naturalness
modeling, and we qualitatively and quantitatively demonstrated thatthe HumanGAN can represent a humans’ perceptually recognizable
distribution. As future work, we address the scalability of the HumanGAN in terms of the number of data and feature dimensions.
Acknowledgements: This research and development was supported by the SECOM Science and Technology Foundation and the
MIC/SCOPE #182103104.

6. REFERENCES[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, Montreal, Canada, Dec. 2014,
pp. 2672–2680.
[2] Y. Saito, S. Takamichi, and H. Saruwatari, “Statistical parametric speech synthesis incorporating generative adversarial
networks,” IEEE/ACM Transactions on Audio, Speech, andLanguage Processing, vol. 26, no. 1, pp. 84–96, Jan. 2018.
[3] M. Mirza and S. Osindero, “Conditional generative adversarial
nets,” arXiv:1411.1784, 2015.
[4] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu, “SeqGAN: Sequence generative adversarial nets with policy gradient,” in Proc. AAAI, San Francisco, U.S.A., Feb. 2017.
[5] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box
adversarial attacks with limited queries and information,” inProc. ICML, Stockholm, Sweden, Jul. 2018, vol. 2, pp. 2137–
2146.
[6] A. J Quinn and BB Bederson, “Human computation: a survey
and taxonomy of a growing field,” in CHI ’11 Proceedingsof the SIGCHI Conference on Human Factors in Computing
Systems, Vancouver, Canada, May 2011, pp. 1403–1402.
[7] Y. Sakata, Y. Baba, and H. Kashima, “CrowNN: Humanin-the-loop network with crowd-generated inputs,” in Proc.
ICASSP, Brighton, U.K., May 2019, pp. 7555–7559.
[8] N. Jaques, J. McCleary, J. Engel, D. Ha, F. Bertsch, R. Picard, and D. Eck, “Learning via social awareness: Improving
a deep generative sketching model with facial feedback,” inarXiv preprint arXiv:1802.04877, 2018.
[9] Y. Koyama, D. Sakamoto, and T. Igarashi, “Crowd-poweredparameter analysis for visual design exploration,” in Proc.
UIST, New York, U.S.A., Oct. 2014, pp. 65–74.
[10] E. Mansimov, E. Parisotto, J. Ba, and R. Salakhutdinov, “Generating images from captions with attention,” in Proc. ICLR,San Juan, Puerto Rico, May 2016.
[11] S. Shinagawa, K. Yoshino, S. Sakriani, Y. Suzuki, and S. Nakamura, “Image manipulation system with natural language instruction,” IEICE Transactions on Information and Systems,
vol. J102-D, no. 8, pp. 514–529, Aug. 2019 (in Japanese).
[12] J.-J. Zhu and J. Bento, “Generative adversarial active learning,”
in arXiv preprint arXiv:1702.07956, 2017.
[13] Y. Deng, K. Chen, Y. Shen, and H. Jin, “Adversarial active
learning for sequences labeling and generation,” in Proc. IJCAI, Stockholm, Sweden, Jul. 2018, pp. 4012–4018.
[14] I. Goodfellow, “NIPS 2016 tutorial: Generative adversarial
networks,” in Proc. NIPS, Barcelona, Spain, Dec. 2016.
[15] “Vowel database: Five Japanese vowels of males, females, andchildren along with relevant physical data (JVPD),” http://
research.nii.ac.jp/src/en/JVPD.html.
[16] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoderbased high-quality speech synthesis system for real-time applications,” IEICE transactions on information and systems, vol.
E99-D, no. 7, pp. 1877–1884, Jul. 2016.
[17] M. Morise, “D4C, a band-aperiodicity estimator for highquality speech synthesis,” Speech Communication, vol. 84, pp.
57–65, Nov. 2016.

[18] A. Lee, T. Kawahara, and K. Shikano, “Julius — an open
source real-time large vocabulary recognition engine,” in Proc.
EUROSPEECH, Aalborg, Denmark, Sep. 2001, pp. 1691–
1694.
[19] “Lancers,” https://www.lancers.jp/.
[20] S. Tokui, R. Okuta, T. Akiba, Y. Niitani, T. Ogawa, S. Saito,
S. Suzuki, K. Uenishi, B. Vogel, and H. Y Vincent, “Chainer:
A deep learning framework for accelerating the research cycle,” in KDD ’19 Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,
Anchorage, AK, U.S.A., Aug. 2019, pp. 2002–2011.