HUMANACGAN: CONDITIONAL GENERATIVE ADVERSARIAL NETWORKWITH HUMAN-BASED AUXILIARY CLASSIFIERAND ITS EVALUATION IN PHONEME PERCEPTION
Yota Ueda1 , Kazuki Fujii2 , Yuki Saito1 , Shinnosuke Takamichi1 , Yukino Baba3 and Hiroshi Saruwatari1

ABSTRACT
We propose a conditional generative adversarial network (GAN) incorporating humans’ perceptual evaluations. A deep neural network(DNN)-based generator of a GAN can represent a real-data distribution accurately but can never represent a human-acceptable distribution, which are ranges of data in which humans accept the naturalness regardless of whether the data are real or not. A HumanGAN was proposed to model the human-acceptable distribution. A
DNN-based generator is trained using a human-based discriminator,i.e., humans’ perceptual evaluations, instead of the GAN’s DNNbased discriminator. However, the HumanGAN cannot represent
conditional distributions. This paper proposes the HumanACGAN,
a theoretical extension of the HumanGAN, to deal with conditional
human-acceptable distributions. Our HumanACGAN trains a DNNbased conditional generator by regarding humans as not only a discriminator but also an auxiliary classifier. The generator is trainedby deceiving the human-based discriminator that scores the unconditioned naturalness and the human-based classifier that scores theclass-conditioned perceptual acceptability. The training can be executed using the backpropagation algorithm involving humans’ perceptual evaluations. Our experimental results in phoneme perception
demonstrate that our HumanACGAN can successfully train this conditional generator.
Index Terms— Generative adversarial network, human computation, conditional generator, auxiliary classifier, black-box optimization, speech perception
1. INTRODUCTION
Deep generative models of machine learning have contributed to media research [1, 2, 3]. A generative adversarial network (GAN) [1]is one of the strongest generative models. It has been applied inspeech modeling [4, 5]. The GAN consists of a set of deep neuralnetworks (DNNs), a generator, and a discriminator. The generator istrained to deceive the discriminator, and the discriminator is trained
to distinguish between real and generated data. After iterating them,
the trained generator represents a real-data distribution and can randomly generate data that follows the real-data distribution.
The GAN cannot represent an outer side of the real-data distribution. However, humans can accept out-sided media as natural. In
speech perception, humans can recognize a voice as a human’s eventhough the voice is out of the real-data distribution (i.e., an actual humans’ voice). For example, that range contains synthesized voices
or processed voices. In this study, we call this data range perceived
by humans the human-acceptable distribution. HumanGAN [6] wasproposed to model the human-acceptable distribution by using humans as the discriminator of the GAN. The top of Fig. 1 shows

Generation Training

Graduate School of Information Science and Technology, The University of Tokyo, Japan.
2
National Institute of Technology, Tokuyama College, Japan.
3
Faculty of Engineering, Information and Systems, University of Tsukuba, Japan.

Generation Training

1

GAN
Generator
Discriminator
Prior
Data
Real/Fake
Backpropagation

HumanGAN
HumansNatural?
Backpropagation

Unconditional human-acceptable distr.
Unconditional real-data distr.
HumanACGAN (proposed)
ACGAN
Real/Fake
Natural?
Belong to class?
Class 1/ 2/ ... ClassClass
Backpropagation
Backpropagation
Class

Class 1 Class 2 ...
Conditional real-data distr.

Class
Class 1 Class 2 ...
Conditional human-acceptable distr.

Fig. 1. Comparison of four GANs. We extend the HumanGAN to
the conditional modeling in the same way the GAN was extended to
the ACGAN. While an ACGAN trains a conditional generator with a
DNN-based discriminator and an auxiliary classifier, the HumanACGAN trains one with a humans-based discriminator and an auxiliary
classifier. The trained generator of the HumanACGAN can represent
human-acceptable distributions conditioned by input class labels.
the comparison of a GAN and HumanGAN. The HumanGAN regards humans as a black-boxed system that outputs a difference inposterior probabilities given generated data. The DNN-based generator is trained using the backpropagation algorithm, including thehuman-based discriminator. The trained generator can represent thehuman-acceptable distribution.
However, the HumanGAN’s generator cannot achieve morepractical generative modeling such as text-to-speech synthesis [7]and voice conversion [8] because it cannot represent conditional
distributions. Because the GAN was extended to the conditional
GAN [9, 10], we expect that the HumanGAN can be extended
to the conditional modeling. Namely, we train the HumanGAN’s
generator conditioned on the desired class label, and it represents
the class-specific human-acceptable distribution as a result. This
will contribute establishing a DNN-based framework to model thetask-oriented perception by humans [11, 12].
In this paper, we propose the HumanACGAN, aiming to train a
DNN-based conditional generator using humans’ perceptual evaluations. To this end, we extend the HumanGAN by introducing
an auxiliary classifier GAN (ACGAN) [10]: class-conditional expansion of a GAN. Fig. 1 shows a comparison of four GANs: a
GAN, an ACGAN, a HumanGAN, and our HumanACGAN. The
ACGAN uses a DNN-based auxiliary classifier to train a conditional
DNN-based generator in addition to a discriminator of the GAN.
Our HumanACGAN replaces both the DNN-based discriminatorand auxiliary classifier with humans. The HumanACGAN’s generator is trained using human-perception-based discrimination andclassification. This training is operated using an expansion of thebackpropagation-based algorithm incorporating humans’ perceptual
evaluations as a discriminator and an auxiliary classifier. We evaluated the HumanACGAN in phoneme perception, a task to train
a DNN-based generator that represents phoneme-specific humanacceptable distributions. The experimental results show that 1)
the phoneme-conditioned human-acceptable distributions are wider
than the real-data ones and that 2) the HumanACGAN can successfully train a generator that represents conditional human-acceptable
distributions.

data: class
Backpropagation using
approximated gradient×R [times]

Humans

Evaluate the difference in
Naturalness:
Class acceptability:

Fig. 2. Generator training process of proposed HumanACGAN. A
human observes two perturbed data and evaluates their perceptual
difference in two views: naturalness and class acceptability. They
give global and class-specific gradients, respectively. Evaluations
and perturbations are used for backpropagation to train the generator.

2. RELATED WORKS
2.1. ACGAN
The ACGAN [10] trains a DNN-based conditional generator that
represents real-data distributions conditioned on class labels. To
achieve this, the ACGAN uses a DNN-based auxiliary classifier in
addition to a DNN-based discriminator of the GAN. The generator G (·) transforms a prior noise z = [z 1 , · · · , z n , · · · , z N ] into
data x̂ = [x̂1 , · · · , x̂n , · · · , x̂N ], conditioned on class labels c =[c1 , · · · , cn , · · · , cN ], i.e., x̂n = G (z n , cn ). N denotes the number of data. The prior noise follows a known probability distribution, e.g., a uniform distribution U (0, 1). Here, let real data be
x = [x1 , · · · , xn , · · · , xN ]. The real data also have the same corresponding class label c. A discriminator DS (·) and a classifier DC (·)
are used for the generator training. The DS (·) takes xn or x̂n as an
input and outputs a posterior probability that the input is real data.
The DC (·) takes an xn or x̂n as an input and outputs a posterior
probability that the input source belongs to each class. Objectivefunctions in training are formulated as
LS =

N
X

log DS (xn ) +

n=1

LC =

N
X

N
X

log (1 − DS (G (z n , cn ))) ,(1)∂ x̂n /∂θ can be estimated analytically, but ∂LS /∂ x̂n cannot because the human-based discriminator DS (·) is not differentiable. The HumanGAN uses the natural evolution strategy(NES) [13] algorithm to approximate the gradient. A small per(r)turbation ∆xn randomly generated from a multivariate Gaussian
distribution N 0, σ 2 I , and it is added to a generated datum x̂n . r
is the perturbation index (1 ≤ r ≤ R). σ, I are the standard deviation and the identity matrix, respectively. Next, a human observes(r)(r)
two perturbed data {x̂n + ∆xn , x̂n − ∆xn } and evaluates thedifference in their posterior probabilities of naturalness:(r)(r)∆DS (x̂(r)n ) ≡ DS (x̂n + ∆xn ) − DS (x̂n − ∆xn ).

with [13]
R


∂LS
1 X
(r)
· ∆x(r)
=
∆D
x̂
S
n .
n
∂ x̂n
2σ 2 R r=1

n=1

log DC (xn , cn ) +

n=1

N
X

log (DC (G (z n , cn ) , cn )) . (2)LS is the objective function of the GAN, and LC enables training theconditional generator. The generator is trained to maximize −LS +λLC , where λ is a hyperparameter.
The ACGAN trains the generator using real data, and the generator represents the real-data distribution conditioned on the dataclass. However, because the human-acceptable distribution is wider
than the real-data distribution [6], the ACGAN cannot represent thefull range of that distribution.
2.2. HumanGAN
The HumanGAN [6] was proposed to represent the human-acceptable
distribution in a wider manner than a real-data distribution. A DNNbased unconditional generator of the HumanGAN is trained using
humans’ perceptual evaluations instead of the DNN-based discriminator of the GAN. The generator G (·) transforms prior noise z n
to data x̂n unconditionally. A human-based discriminator DS (·) isdefined to deal with humans’ perceptual evaluations. DS (·) takesx̂n as an input and outputs a posterior probability that the input isperceptually acceptable. The objective function is
LS =

(5)

3. HUMANACGAN

n=1

N
X

(4)

(r)
∆DS (x̂n ) ranges from −1 to 1. For instance, a human will answer(r)(r)∆DS (x̂n ) = 1 when he/she perceives that x̂n +∆xn is substan(r)tially more acceptable than x̂n − ∆xn . ∂LS /∂ x̂ is approximated

3.1. Training
We propose the HumanACGAN. As shown in Fig. 2, the HumanACGAN has a conditional generator that represents classspecific human-acceptable distributions, while the HumanGAN
only deals with one class. As well as the ACGAN, our HumanACGAN consists of a conditional generator, a discriminator, and an
auxiliary classifier. The DNN-based generator G (·) is the sameas that of the ACGAN; it transforms prior noise z n into data x̂nconditioned on a class label cn . The discriminator and the auxiliary
classifier are redefined to incorporate humans’ perceptual evaluations. The human-based discriminator DS (·) is the same as that of
the HumanGAN; it outputs a posterior probability of global (i.e.,class-independent) naturalness. In addition, the human-based auxiliary classifier DC (·) evaluates class acceptability, whether or not
the data can belong to the class cn . DC (·) takes x̂n generated from
G (·) and class label cn as inputs and outputs a posterior probability
that the input is perceptually acceptable as the class”. The objectivefunctions areNXLS =DS (G (z n , cn )) ,
(6)
n=1

LC =
DS (G (z n )) .

(3)

n=1

The generator is trained to maximize LS . A model parameter θ ofG (·) is iteratively updated as θ ← θ + α∂LS /∂θ, where α is thelearning coefficient, and ∂LS /∂θ = ∂LS /∂ x̂n · ∂ x̂n /∂θ.

NX

DC (G (z n , cn ) , cn ) .

(7)

n=1

A model parameter θ of G (·) is estimated by maximizing LS +λLC .
θ is iteratively updated as follows.
∂ (LS + λLC )θ ←θ + α
,
(8)
∂θ



∂ (LS + λLC )
∂ x̂n
∂LS
∂LC
·
=
+λ
.
(9)
∂θ
∂ x̂n
∂ x̂n
∂θ
As well as the HumanGAN, ∂ x̂n /∂θ can be estimated analytically,but ∂LS /∂ x̂n and ∂LC /∂ x̂n cannot. We formulate gradient approximation using the NES algorithm. A human observes two per(r)(r)
turbed data {x̂n + ∆xn , x̂n − ∆xn } and evaluates two kindsof difference in their posterior probabilities. The first is the sameas the HumanGAN, i.e., the human evaluates “to what degree theinputs are perceptually different in the view of naturalness” and an(r)swers ∆DS (x̂n ) for approximating ∂LS /∂ x̂n as in Eq. (4). The
second is for the class-specific question, “to what degree the inputs
are different in the view of class acceptability.” The difference in the(r)posterior probability ∆DC (x̂n , cn ) is defined as(r)∆DC (x̂(r)x̂n + ∆xn
, cn
n , cn ) ≡DC(r)− DC x̂n − ∆xn, cn .
(10)

Fig. 3. Color maps representing posterior probabilities of naturalness and class acceptability (“accep.”) of /i/ and /e/.

In the same way as the HumanGAN, ∂LS /∂ x̂n can be approximated(r)using ∆DS (x̂n ). Unlike ∆DS (·), ∆DC (·) becomes the class(r)specific difference. Namely, ∆DC (x̂n , cn ) will be 1 when the(r)human perceives that x̂n + ∆xn is substantially more acceptable(r)(r)
than x̂n − ∆xn as the presented class. ∆DC (x̂n , cn ) ranges
from −1 to 1, and ∂LC /∂ x̂ is approximated with
R


∂LC
1 X
(r)
(r)
=
∆DC x̂n, cn · ∆xn
.
(11)
2
∂ x̂n
2σ R r=1
Note that the HumanACGAN does not explicitly involve classification problems. In other words, it only needs to estimate the gradientof multiclass probability function (i.e., the softmax function used in
an ACGAN) as the degree of class-specific acceptability.
3.2. Limitations
The HumanGAN [6] suffers from a mode collapse problem [14], agradient vanishing, and scalability to the data size and dimensionality. These problems still remain in our HumanACGAN. Therefore,
our experiments followed the initialization way and data preprocessing of the paper [6]. We first estimated histograms of the posterior
probabilities and initialized θ by referring to the histograms. Also,
we reduced the data dimensionality using principal component analysis (PCA).
4. EXPERIMENTAL EVALUATIONThis section describes an experimental test of the effectiveness of the
HumanACGAN using Japanese phonemes as classes.
4.1. Experimental setup
Two phonemes we used were /i/ and /e/ of Japanese vowels. We
basically followed the experimental setup of the HumanGAN paper [6]. The used data consisted of 199 female speakers’ utterances recorded in the JVPD [15] corpus. Before extracting speechfeatures, the speech waveforms were downsampled at 16 kHz, andtheir powers were normalized. 513-dimensional log spectral envelopes, fundamental frequency (F0), and aperiodicities (APs) wereextracted every 5 ms from the speech waveforms using the WORLD
vocoder [16, 17]. We extracted the speech features of the vowels/i/ and /e/ using phoneme alignment obtained by Julius [18]. We applied PCA to the log spectral envelopes and used the first and second
principal components. The two-dimensional principal components

Fig. 4. Generated data (points) and gradient (arrows) estimated by
our algorithm. “accep.” denotes acceptability. The gradients of theupper left figure are the weighted sum of the other three gradients.
We can see that the gradients for naturalness and class acceptability
are respectively pointing to darker (i.e., higher posterior) zones.
were normalized to have zero-mean and unit-variance. The speech
waveforms to be evaluated were synthesized using features obtained
from a DNN-based conditional generator in the following way. First,
the first and second principal components were generated by the generator and de-normalized. For the other features, i.e., the F0 and theAPs, we used the average of all speakers. These corresponded to the
speech features of one frame (5 ms). Next, we copied the features for
200 frames to make the perceptual evaluations easy and synthesized
1 second of speech waveforms using the WORLD vocoder.
4.2. Difference between real-data and human-acceptable distributionsFirst, we confirmed that the human-acceptable distributions were
wider than the real-data distributions. We carried out two tests:
evaluations of humans’ tolerance of naturalness and class acceptability. In this experiment, class acceptability denotes whether the datasounded like the presented phoneme. We split the two-dimensional
space into grids and generated a speech waveform for every grid.
Then, we presented a speech waveform to a listener, and the listener
rated the naturalness on a 5-point scale from 1 (bad) to 5 (excellent). Next, we presented a speech waveform and a phoneme label

Fig. 5. Data generated from the initialized or trained generator. The
colors of the data points correspond to those of the color maps forclass acceptability in Fig. 3. Based on posterior probabilities shownin Fig. 3, we can say that the training makes the data move to havehigher posterior probabilities.

Fig. 6. Posterior probability of data generated from initialized(“Init”) or trained (“Trained”) generators. The boxes indicate first,
second (i.e., median), and third quantiles. The line plot indicatesmean value.
(/i/ or /e/) to a listener, and the listener rated the class acceptabilityon the same scale. One-through-five of the obtained scores corresponded to 0.00, 0.25, 0.50, 0.75, and 1.00 of the posterior probability (DS (x̂n ) or DC (x̂n , cn )), respectively. We used the Lancers
crowdsourcing platform [19] to execute the evaluations. The posterior probabilities were averaged for each grid. Each grid was scoredby at least five listeners. The total number of listeners was 105.
Fig. 3 shows the results. As described in Section 4.1, thereal data ware normalized to have zero-mean and unit-variance, and
the ACGAN represents this range. However, as shown in Fig. 3,
the human-acceptable distributions, i.e., darker zones of the colormaps, were wider than the real-data distribution for both naturalness and class acceptability. Therefore, we obtained support for the
HumanACGAN being able to represent this distribution, which an
ACGAN cannot adequately represent.

determining the ability to classify. We empirically set the hyperparameter λ = 2 and used the gradient descent method with a learning
rate of α = 0.0005 for the training. We used Chainer [20] for the
implementation. The number of generated data N , the number of
perturbations R, the number of training iterations, and the standard
deviation of NES σ were set to 50, 5, 4, and 2.0, respectively. We
carried out two perceptual evaluations during the HumanACGAN
training. First, we presented two speech waveforms to a listener, andthe listener rated which one was more natural on a 5-point scale: 1:
the first one, 3: equal, and 5: the second one. Next, we presented two
speech waveforms and one phoneme label (/i/ or /e/) to a listener,
and the listener rated which one sounded like the presented phonemeon the same scale. One-through-five of the obtained scores corresponded to 1.0, 0.5, 0.0, −0.5, and −1.0 of the difference in theposterior probabilities (∆DS (x̂n ) or ∆DC (x̂n , cn )), respectively.
Fig. 4 shows the gradient of every data point. The posterior
probability of Fig. 3 is also drawn for reference, but note that wenever used it during the training. We can say that the gradient byboth naturalness and class acceptability points to each darker range.
This qualitatively indicates that the gradient shown in Eqs. (5), (11)was properly estimated. Fig. 5 shows the data generated from initialized and trained generators. On the basis of posterior probabilities
shown in Fig. 3, we can say that the training makes the data move sothat they have higher posterior probabilities of both naturalness andclass acceptability. This qualitatively indicates that the loss functions shown in Eqs. (6), (7) improved the generator to represent theconditional human-acceptable distributions.
4.4. Increase in posterior probabilities during training
Finally, we quantitatively verified that the training increases theposterior probabilities of naturalness DS (·), and class acceptability
DC (·). We prepared two types of data: closed and open. The closed
data were generated from the prior noise used while the training,and open data was generated from a newly sampled prior noise thatwas not used during the training. The posterior probabilities of theclosed/open data were scored in the same manner as the ones in
Section 4.2. The total number of listeners was 160.
Fig. 6 shows the box plots of the posterior probability. The
training iteration increased the posterior probabilities of both naturalness and class acceptability with not only the closed data but also
the open data. Therefore, we can say that our training can increase
the objective values consisting of posterior probabilities and that thegenerator of the HumanACGAN can represent conditional humanacceptable distributions.

4.3. Transition of generated data during training
Next, we executed the HumanACGAN training and qualitatively
evaluated the generated data and approximated gradient. The priornoise followed a two-dimensional uniform distribution U (0, 1). We
randomly generated this prior noise before the training and fixed it
during the iterations. Half of the data belonged to the class label/i/, and the remaining data belonged to /e/. The generator was asmall feed-forward neural network consisting of a two-unit input
layer, 2 × 4-unit sigmoid hidden layers, and a two-unit linear output
layer. The output layer was conditioned using the two dimensional
one-hot class label vector. We performed random initialization of
the generator until these conditions were satisfied: The data generated from the initialized generator should cover ranges of the higher
posterior probabilities of naturalness and class acceptability, and thedata of each class should leak into the other class distribution for

5. CONCLUSION
We proposed the HumanACGAN, which can conditionally represent
humans’ perceptually acceptable distributions. We reconfigured a
discriminator and an auxiliary classifier of an ACGAN to utilize humans’ perceptual evaluations. The DNN-based conditional generator of the HumanACGAN was trained using two kinds of human’s
perceptual evaluations: naturalness and class acceptability, whichwere used for the discriminator and the auxiliary classifier, respectively. We evaluated the effectiveness of the HumanACGAN using
qualitative and quantitative experiments. We are planning to expand
the scalability of the HumanACGAN in terms of scalability to thedata size and dimensionality as part of our future work.
Acknowledgements: Part of this work was supported by the
MIC/SCOPE #182103104.

6. REFERENCES[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, Montreal, Canada, Dec. 2014,
pp. 2672–2680.
[2] D. Kingma and M. Welling, “Auto-encoding variational
bayes,” arXiv, vol. abs/1312.6114, 2013.
[3] L. Dinh, D. Krueger, and Y. Bengio, “NICE: Non-linear independent components estimation,” in Proc. ICLR, San Diego,
U.S.A., May 2015.
[4] Y. Saito, S. Takamichi, and H. Saruwatari, “Statistical parametric speech synthesis incorporating generative adversarial
networks,” IEEE/ACM Transactions on Audio, Speech, andLanguage Processing, vol. 26, no. 1, pp. 84–96, Jan. 2018.
[5] Y. Hono, K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda,“Singing voice synthesis based on generative adversarial networks,” in proc. ICASSP, Brighton, United Kingdom, May
2019, pp. 6955–6959.
[6] K. Fujii, Y. Saito, S. Takamichi, Y. Baba, and H. Saruwatari,“HumanGAN: generative adversarial network with humanbased discriminator and its evaluation in speech perception
modeling,” in Proc. ICASSP, Barcelona, Spain, May 2020,
pp. 6239–6243.
[7] Y. Sagisaka, “Speech synthesis by rule using an optimal selection of non-uniform synthesis units,” in Proc. ICASSP, New
York, U.S.A., Apr. 1988, pp. 679–682.
[8] Y. Stylianou, O. Cappé, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Transactions on
Speech and Audio Processing, vol. 6, no. 2, pp. 131–142, Mar.
1998.
[9] M. Mirza and S. Osindero, “Conditional generative adversarial
nets,” arXiv preprint arXiv:1411.1784, 2014.
[10] A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with auxiliary classifier GANs,” in Proc. ICLR, Vancouver,
Canada, Apr. 2018.
[11] C. Chiu, Y. Koyama, Y. Lai, T. Igarashi, and Y. Yue, “Humanin-the-loop differential subspace search in high-dimensional
latent space,” ACM Transactions on Graphics (TOG), vol. 39,
no. 4, pp. 85–1, 2020.
[12] J. Peterson, J. Suchow, K. Aghi, A. Ku, and T. Griffiths, “Capturing human category representations by sampling in deep
feature spaces,” arXiv preprint arXiv:1805.07644, May 2018.
[13] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box
adversarial attacks with limited queries and information,” inProc. ICML, Stockholm, Sweden, Jul. 2018, vol. 2, pp. 2137–
2146.
[14] I. Goodfellow, “NIPS 2016 tutorial: Generative adversarial
networks,” in Proc. NIPS, Barcelona, Spain, Dec. 2016.
[15] “Vowel database: Five Japanese vowels of males, females, andchildren along with relevant physical data (JVPD),” http://
research.nii.ac.jp/src/en/JVPD.html.
[16] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoderbased high-quality speech synthesis system for real-time applications,” IEICE transactions on information and systems, vol.
E99-D, no. 7, pp. 1877–1884, Jul. 2016.

[17] M. Morise, “D4C, a band-aperiodicity estimator for highquality speech synthesis,” Speech Communication, vol. 84, pp.
57–65, Nov. 2016.
[18] A. Lee, T. Kawahara, and K. Shikano, “Julius — an open
source real-time large vocabulary recognition engine,” in Proc.
EUROSPEECH, Aalborg, Denmark, Sep. 2001, pp. 1691–
1694.
[19] “Lancers,” https://www.lancers.jp/.
[20] S. Tokui, R. Okuta, T. Akiba, Y. Niitani, T. Ogawa, S. Saito,
S. Suzuki, K. Uenishi, B. Vogel, and H. Y Vincent, “Chainer:
A deep learning framework for accelerating the research cycle,” in Proc. KDD, Anchorage, U.S.A., Aug. 2019, pp. 2002–2011.