Unifying Human and Statistical Evaluation for
Natural Language Generation
Tatsunori B. Hashimoto*1,2 Hugh Zhang*1 Percy Liang1,2(* equal contribution)
1
Department of Computer Science 2 Department of Statistics
Stanford University{thashim,hughz}@stanford.edu

Abstract
How can we measure whether a natural language generation system produces both highquality and diverse outputs? Human evaluation captures quality but not diversity, as it
does not catch models that simply plagiarize
from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity but not quality, as models that occasionally emit low quality samples would be insufficiently penalized. In this paper, we propose a unified framework which evaluates bothdiversity and quality, based on the optimal error rate of predicting whether a sentence ishuman- or machine-generated. We demonstrate that this error rate can be efficiently estimated by combining human and statisticalevaluation, using an evaluation metric which
we call HUSE. On summarization and chitchat dialogue, we show that HUSE detects diversity defects which fool pure human evaluation and that techniques such as annealing
for improving quality actually decrease HUSE
due to decreases in diversity.

1

Introduction

Generating text is a core part of many NLP tasks
such as image captioning (Lin et al., 2014), opendomain dialogue (Sordoni et al., 2015), story generation (Roemmele, 2016), and summarization(Nallapati et al., 2016). However, proper evaluation of natural language generation has provendifficult (Liu et al., 2016; Novikova et al., 2017;
Chaganty et al., 2018). A good evaluation metricshould not only capture the quality of generation,
but also the diversity of generation, which is especially crucial for open-ended tasks like dialogue orstory generation.
Human evaluation, which is often viewed as
the gold standard evaluation, captures quality perfectly but fails to capture diversity. As an example, for language modeling, a model that di-pliang@cs.stanford.edu

Reference

Reference
Cleared coach facing anothergrilling from British swim bosses
Agassi withdraws from Australian open

Model Generations
Agassi bows out of Australian open
Sharon has stroke for stroke

Model Probability (pmodel)

Figure 1. HUSE is twice the classification errorof distinguishing reference and generated text represented as (human judgment, pmodel ) pairs. This identifies samples with defects in both quality (Sharon
has stroke . . .) and diversity (Cleared coach facing. . .).

rectly plagiarizes sentences from the training set
would pass the human quality bar but would havezero generalization ability and thus have inadequate diversity. On the other hand, statistical evaluation—i.e., perplexity on a test set—captures diversity, as it ensures a model must have someprobability of generating novel sentences, but perplexity provides an inadequate measure of quality (Theis et al., 2015): modifying a perfect modelby making it incapable of generating just a single reference sentence results in infinite perplexity even though the model has near perfect diversity and sample quality. Automatic metrics such
as BLEU (Papineni et al., 2002) and ROUGE (Linand Rey, 2004) capture quality better than perplexity but still correlate poorly with human evaluation, the gold standard for judging sample quality (Novikova et al., 2017; Chaganty et al., 2018);
moreover, these metrics do not capture diversity.
We propose defining the ideal evaluation metric as twice the error rate of the optimal discriminator that classifies whether a sentence is generated from the reference distribution or from themodel (Section 2). If a model generates gibberish(low quality), the optimal discriminator can classify these accurately as coming from the model.

If the reference distribution contains sentences themodel cannot generate (low diversity), the optimal
discriminator can classify these accurately as coming from the reference.
Unfortunately, the optimal discriminator is unavailable. Human discriminators cannot capture
diversity effectively, and learned discriminators—e.g., from a Generative Adversarial Network(Goodfellow et al., 2014) or one trained on humanjudgments (Lowe et al., 2017)—are too unreliable
to use as a replacement for human evaluation.
Our key result (Section 3) is that the optimalclassifier depends only on two numbers: the probability of a sentence under the model and the probability under the reference distribution. The former can be computed from the model, and the latter can be well approximated by human judgmentscores. The resulting two-dimensional space is illustrated in Figure 1. We apply a simple k-nearest
neighbor classifier to this space and define Human Unified with Statistical Evaluation (HUSE)
as twice the leave-one-out error of this classifier.
We apply HUSE to four natural language generation tasks (Section 5): language modeling, dialogue, story generation, and summarization. First,
we show that human evaluation alone is insufficient to discriminate model generations from thereferences, leading to inflated estimates of model
performance. In contrast, HUSE is able to reveal
gaps between current models and true human performance. We also show that techniques for improving sample quality such as annealing actually
increase distinguishability between the model andreference due to substantial losses in diversity.

2

Optimal Discriminator

Consider a natural language generation task where
the model is given a context x (e.g., a dialogue
history) drawn from some p(x) and must output a
distribution over possible sentences pmodel (y | x).
We define an idealized evaluation metric based onwhether pmodel is close to a reference distribution
pref , which is generally human-generated.1 Specifically, consider a random variable y drawn fromeither the reference or the model based on an indi1

While some tasks care only for quality and thus only
require pmodel to place mass on some high quality y, we demand pmodel to place mass on all high quality y. This diversity
is important for open-ended tasks such as dialogue or story
generation.

cator z ∼ Bernoulli( 21 ):(pref (y | x)
y | x, z ∼pmodel (y | x)

if z = 1
if z = 0.

(1)

Define L∗ to be twice the lowest possible error
over any discriminator f that attempts to determine z based on x and y:defL∗ = 2 inf P[f (x, y) 6= z].
f(2)L∗ measures similarity between pmodel and pref ; itis 0 if pmodel and pref are disjoint and 1 if they are
identical.2
Obstacles. Unfortunately, L∗ is unattainable because it requires computing the optimal discriminator. In the spirit of the Turing Test, we could
consider using the error rate of a human discriminator fhum as a proxy. However, existing humanevaluation requests sentences judgments one at atime, resulting in fhum having knowledge of pref
but not pmodel . Thus, fhum is unable to determine
which sentences a model cannot generate.
As a concrete example, suppose pref placed auniform distribution over some set S. Without
knowledge of pmodel the most sensible discriminator is to predict z = 1 (reference) when y ∈ S.
This discriminator achieves the same classification
error of 0.5 for both the perfect model pmodel =pref and one which can only return a single y ∈ S.
We could try to reveal pmodel to humans by showing multiple samples simultaneously, but this isexpensive – and as we will later see – unnecessary.
Another option is to learn f over an expressiveclass of functions such as neural networks on data
sampled from pmodel and pref . This is analogous tolearning the discriminator in a Generative adversarial network (GAN) (Goodfellow et al., 2014)or an evaluation metric from human judgments(Lowe et al., 2017). However, as (x, y) are highdimensional objects, training a good classifier isextremely difficult (and perhaps not significantly
easier than solving the original generation problem). Indeed, learned evaluation metrics do not
generalize very well (Lowe et al., 2017; Chaganty
et al., 2018). Unlike these approaches which seek
to replace human evaluation, our focus will instead
be on combining human and automatic statisticalevaluation to estimate the optimal classifier error.
Note that L∗ is a linear function of todeftal
kpmodel − pref kTV=
P variational distance:
∗
p(x)
|p
(y
|
x)
−
p
(y
|
x)|
=
(1
−
L
).
model
ref
x,y
See Appendix A.1 for details.
2Human Unified with Statistical
Evaluation (HUSE)

3.2

Our key result is that the optimal discriminator depends on (x, y) only through a two-dimensionalsufficient statistic (Section 3.1), motivating an approximation which we call HUSE (Section 3.2).
We finish by showing how the various evaluation
metrics relate (Section 3.3).
Firstly, for any feature map φ that maps (x, y) toφ(x, y) ∈ Rd , define an evaluation score L(φ) to
be twice the error rate of the optimal discriminatorthat depends on (x, y) through φ:def

L(φ) = 2 inf P[f (φ(x, y)) 6= z].
f(3)

Note that the evaluation score L(φ) given bya feature map φ optimizes over all functions that
depend on φ (3). Thus, the more information φcontains, the lower L(φ) is. This has two implications: First, any feature map φ yields an (optimistic) upper bound on L∗ , meaning that L(φ) can
detect when a model is poor but cannot certify that
it is good. Second, adding features to φ can only
improve this bound.
3.1

While we can directly compute pmodel (y | x) for
many probabilistic models, pref (y | x) is unattainable, so L(φopt ) is not computable. However, thewisdom of the crowds (Surowiecki, 2004; Ungar
et al., 2012) suggests that pooling together the
judgments of many humans can often produce surprisingly reliable estimates of real-world probabilities such as pref (y | x), even if no individual
human is particularly reliable. With this motivation, we ask Amazon Mechanical Turk workers torate a sentence from 1–5 based on how “typical”
it is (see Appendix A.2 for more details). We define HJ(x, y) to be the average response over 20
crowdworkers. Figure 2 shows that for a language
modeling task on the Reddit corpus,3 HJ(x, y)strongly correlates with the actual log-frequencyof y in the corpus. The high correlation suggeststhat human judgments HJ(x, y) are a good surrogate for log pref .

Two features suffice

Let us consider the following two-dimensionalfeature map:def

φopt (x, y) = [pref (y | x), pmodel (y | x)] .

Proposition 1. The two-dimensional feature map
φopt achieves the optimal discriminator score:L(φopt ) = L∗ .
Proof We compute the true posterior over z
given x, y. Since p(z = 1) = p(z = 0) = 12 ,p(y | x, z = 1) = pref (y | x) and p(y | x, z =0) = pmodel (y | x), by Bayes’ rule:p(z = 1 | x, y) =pref (y | x).
pref (y | x) + pmodel (y | x)

The optimal discriminator simply predicts z = 1
if pref (y | x) > pmodel (y | x) and z = 0
otherwise. In other words, the decision line is
φopt (x, y)1 > φopt (x, y)2 .

5
4
3
2
1

101
105
103
Frequency in Reddit corpus (log scale)Figure 2. On the Reddit corpus, human judgment(4)

From the arguments above, it is clear that
L(φopt ) ≥ L∗ , but perhaps somewhat surprisingly,
we actually have equality:

HUSE features

Human Judgment (HJ) Score

3(HJ) of the “typicality” of a sentence y correlates
strongly (r = 0.92) with its frequency in the corpus,
suggesting that HJ is a good surrogate for log pref .
Error bars at the 90% confidence interval.

In addition, we found that rather than using
the model probability pmodel (y | x) directly asa feature, normalizing by sentence length len(y)yielded lower (tighter) scores, and since any feature map yields a valid upper bound on L∗ , wedefine the following feature map:
def log pmodel (y | x)
φhuse (x, y) =, HJ(x, y) (5)len(y)
which is used to define the (population) HUSE
score L(φhuse ).
3.3

Guarantees derived from HUSE

We now show that the HUSE score satisfies two
nice properties: (i) a model with low HUSE score
3

We used the Reddit corpus due to crowdworker familiarity, corpus size, and short average sentence length, which
results in a wide range of sentence frequencies.

must have low optimal classification error (i.e.
HUSE can detect low-quality models) and (ii) a
model with high HUSE score must have high classification error when using humans as classifiers.
Consider a feature map that only includes hudef
man evaluation: φhj (x, y) = [HJ(x, y)]. Because φhuse also incorporates human evaluation,L(φhuse ) is always tighter than the human discriminator error L(φhj ):Proposition 1 (Relationship between HUSE, human evaluation, and optimal scores).
L(φhj ) ≥ L(φhuse ) ≥ L∗ .

Evaluating models with HUSE

In this section, we show how we can estimate
the population HUSE score L(φ) from finite data
(Section 4.1). We then show how HUSE canbe decomposed into a score that measures quality (HUSE-Q) and a score that measures diversity (HUSE-D), which allows us to study qualitydiversity tradeoffs (Section 4.2).
4.1

Algorithm 1 Computing discriminator score
Require: Feature map φContexts x1 , . . . , xn
Reference outputs y1 , . . . , yn
Model outputs y10 , . . . , yn0
1: Construct dataset:D=(6)Furthermore, the main difference between
L(φhuse ) and L∗ is that the former uses HJ(x, y)and the latter uses pref . But as we argued using
Figure 2, HJ(x, y) is strongly correlated with pref ,so we expect L(φhuse ) to be relatively close to L∗ .

4

vectors φ(x, y) scaled componentwise to have unit
variance. The overall procedure for computing theestimate L̂(φ) is formally defined in Algorithm 1.

Learning a discriminator

For any feature map φ, we show how to produce
an estimate of L(φ). Fix n contexts x1 , . . . , xn .
First, we draw n examples y1 , . . . , yn from the reference distribution pref (y | x), which are usuallyhuman-generated sentences from a test set. We
also draw n examples y10 , . . . , yn0 from the model
pmodel (y | x) we wish to evaluate. Next, for eachof the 2n examples (x, y), we compute the feature
map φ(x, y), which might involve evaluating themodel probability pmodel (y | x) as well as collecting human judgments HJ(x, y) from crowdworkers.
Finally, we compute the leave-one-out error ofa classifier that tries to predict whether a given example (x, y) comes from the reference distribution(z = 1) or the model (z = 0). The choice ofclassifier is not important, but we chose k-nearest
neighbors because it is simple, requires no training, and can capture arbitrary continuous decisionboundaries. Specifically, we set k = 15 and define neighbors using L2 distances over the featuren[{(φ(xi , yi ), 1), (φ(xi , yi0 ), 0)}

i=1
def

2: L̂(φ) = leave-one-out error of k-NN on D

4.2

Quality-diversity decomposition

We now define the (empirical) HUSE score using
the feature map φhuse :def

HUSE = L̂(φhuse ).

(7)

We define the quality component of HUSE
(HUSE-Q) similarly using human judgments
alone:
def

HUSE-Q = L̂(φhj ).

(8)Since humans alone can detect quality defects
in a model, any increase in error from removing
pmodel must come from a model’s lack of diversity. Therefore, we define the diversity component(HUSE-D) as follows:def

HUSE-D = 1 + HUSE − HUSE-Q,(9)

which implies the decomposition (1−HUSE-D)+(1 − HUSE-Q) = 1 − HUSE. As long as
the discriminators are non-trivial (obtaining better
than chance performance with sufficient data), allscores are contained in [0, 1]. Here, HUSE-D = 1
implies that the model suffers no diversity defects,
while HUSE-D = 0 indicates that the examples
could be discriminated perfectly due to a lack ofdiversity.

5

Experiments

5.1

Experimental setup

We use HUSE to evaluate three different types ofsingle-sentence natural language generation tasks:

Method

Summarizationt = 1.0 t = 0.7

Story generationt = 1.0 Retrieval

Chit chat dialoguet = 1.0 t = 0.7

LMt = 1.0

HUSE
HUSE-Q
HUSE-D

0.53
0.58
0.95

0.06
0.15
0.91

0.56
0.56
1.00

0.86
0.88
1.02

0.26
0.92
0.34

0.00
0.47
0.53

0.49
0.92
0.57

Table 1. Performance achieved by the best models on the four tasks, as measured by overall goodness-of-fit (HUSE),
sample quality (HUSE-Q) and diversity (HUSE-D). The scale ranges from 0.0 (completely distinguishable) to 1.0(indistinguishable from reference) where the implied classification error is HUSE/2.

Diversity (HUSE-D)

Quality (HUSE-Q)(i) unconditional and high entropy (language modeling); (ii) conditional and high entropy (story
generation, chit-chat dialogue); and (iii) conditional and low entropy (summarization).
In more detail, the four tasks are:
• Summarization: Giganews story to headline dataset and the pre-trained model from
Gehrmann et al. (2018).
• Story generation: Last sentence generation for ROC stories (Mostafazadeh et al.,2016) using a standard OpenNMT modelwith global attention (Klein et al., 2017).
• Language modeling: One billion word
benchmark language model from Jozefowicz
et al. (2016).
• Chit-chat Dialogue: Two-turn chit-chat dialogue dataset constructed from Reddit comments and their replies (Appendix A.3). We
train a convolutional model from fairseq(Gehring et al., 2017).
For each task, we evaluate three standard
schemes for managing diversity-quality tradeoffs.
We exclude beam search since HUSE ≈ 0 due toits extreme lack of diversity.

Figure 3.

Tradeoffs between HUSE-D and
HUSE-Q for various tasks. Closer to the top right isbetter, and shaded gray lines indicate HUSE. Generation mechanisms can trade-off between diversity
and quality but cannot easily increase the underlying
model performance (HUSE).

We selected several schemes for each task basedon whether we expected them to improve either
HUSE-Q or HUSE-D. For cost reasons, we did
not comprehensively measure all combinations oftasks and generation strategies, but our evaluations
cover the general range of available diversityquality tradeoffs.
Finally, we collect human judgments HJ(x, y)
as per Section 4.1 where we query 20 Amazon
Mechanical Turk crowdworkers for typicality ratings on 100 reference and model sentences.
5.2• Temperature annealing: for any probabilistic model that generates words sequentially,
we sample a word proportional to p1/t where
p is the model’s distribution and t is the temperature parameter.
• Retrieval: using Apache solr, we retrieve
responses from the training set using the default BM25 similarity metric.
• Overfitting: models were trained for triple
the number of minibatches necessary to minimize validation loss. This forces the model
to aggressively memorize the training set andincrease generation quality.

Overall results

The HUSE scores across the four tasks vary
widely. Table 1 shows that language models arenearly indistinguishable, with HUSE = 0.86 andimplied discriminator error of 43%.
In contrast, both summarization and dialogue
are highly distinguishable (HUSE ≈ 0.5) withrelatively low quality when sampled from t =1.0. Human evaluation alone (HUSE-Q) would
suggest that using temperature annealing to emphasize high-probability outputs substantially improves the model (t = 0.7). However, we find that
this increase in sample quality comes at the costof diversity. Examining the achievable HUSE anddiversity tradeoffs in Figure 3 shows that mech-

T=1.0

T=0.9

T=0.7

Figure 4. The two dimensional classification problem in Algorithm 1 on the summarization task with different
softmax temperatures (three panels). Each point represents a sentence (φhuse (xi , yi )), color is the source of the sentence(z), shading is the classification confidence of the nearest neighbor classifier.

5.3

Understanding HUSE

Since HUSE is estimated from a two-dimensionalclassification problem, we can directly visualize the classification problem to understand thediversity-quality tradeoffs.
Figure 4 shows φhuse (xi , yi ) for the summarization task on both reference (blue square) andmodel (red circle) outputs. The shaded areas indicate the decision boundary of the k-nearest neighbor classifier.
At temperature t = 1.0, we find that the classification boundary is mostly horizontal, implying that human judgment can distinguish model
outputs from references. However, there exists acluster of sentences with high HJ and high pmodel
which are essentially indistinguishable. Examining the samples in this top-right region reveals thatthese are news stories with short headlines suchas “nadal pulls out of sydney international” which
can be reliably generated even at t = 1.0.
At lower temperatures of t = 0.9 and t = 0.7

Stability of discriminator accuracy
Discriminator accuracy

anisms such as annealing which improve sample quality actually degrade HUSE due to severe
losses in diversity.
We find that all schemes and models are inadequate for story generation on ROC stories. The
original model (t = 1.0) is very easily distinguishable by a human (HUSE-Q = 0.15), corresponding to a discriminator error of 7%. The retrieval
models can improve this to HUSE-Q = 0.47, but
this comes at the expense of a catastrophic loss ofdiversity. Even the best scheme we found (overfitting the model) could not avoid this tradeoff.
Finally, we observe that directly sampling themodel (t = 1.0) is always diverse. This suggeststhat human evaluation is an appropriate evaluation
for systems which are trained to maximize loglikelihood and generate via sampling.

Figure 5. Estimates of HUSE are robust to small
test set size, but generally require ≈ 20 replicate
measurements for HJ.

the boundary shifts towards becoming diagonal.
Although the distribution is no longer directly separable on human judgment, the two distributions
are quite separable with the inclusion of pmodel .
There is no longer a group of un-identifiable points
on the top right, as the model-generated text now
receives noticeably higher pmodel than the references.
Using Figure 4, we can identify individual examples which were correctly and incorrectly classified based on log p and HJ. Table 2 shows examples of both quality failures and diversity failures
identified by HUSE. For example, the “Diversity
failure” table shows that the summarization model
does not understand that “front office” is a way
to refer to the president and general manager andthus assigns very low probability to the reference.
Improving these models on the diversity failures
will require that the model understand these more
subtle paraphrases. We can also identify model
successes, where the model outputs are truly indistinguishable from the reference, and those inwhich the reference itself is low quality.
5.4

HUSE stability

Recall that HUSE depends on HJ(x, y) which reduces noise by averaging over crowdworkers. We
show that depending on the task, fewer replicate

log pmodel

Quality failure

HJ

Context:

two new vaccines have been shown effective against rotavirus, which is responsible for ahalf-million infant deaths in poor countries each year, research studies published wednesday said.

Model

two new vaccines in the poor countries were effective against go-it-alone study says

-2.3

2.6

Referencenew vaccines for key UNKNOWN virus shown effective

-4.0

4.3

Context:

the buffalo bills sacked tom donahoe as president and general manager on wednesday,
fulfilling expectations of a shake-up after another failure to make the national footballleague playoffs.

Modelbills sack UNKNOWN as president gm and general manager

-0.9

4.3

Reference

nfl ’s bills shake up front office

-5.1

4.3

Diversity failure

Model is indistinguishable
Context:

us veteran and eight-time grand slam winner andre agassi has withdrawn from this month ’s
australian open due to a nagging ankle injury , his management team announced thursday .

Modelagassi bows out of australian open after injury

-1.4

5.3

Reference

agassi withdraws from australian open

-0.3

4.9

Context:

israeli prime minister ariel sharon was undergoing an emergency operation thursday after
suffering a massive stroke.

Modelsharon undergoing emergency operation

-0.8

4.9

Reference

timeline of sharon era

-4.7

2.9

Model outperforms reference

Table 2. Summarization examples extracted from Figure 4 for the OpenNMT pretrained model (t = 1.0) andreferences.

crowdworkers could be used to estimate HJ.
Figure 5 shows the result of subsampling ouroriginal data on 200 sentences and 20 crowdworkers and estimating HUSE. First, we find that using
50 test set examples (Figure 5, left) is sufficient
to give accurate estimates of HUSE. Next, we findthat the necessary replicate count depends heavily
on the task. For easily distinguishable tasks (story
generation), 10 replicates suffice, while less distinguishable tasks (summarization) require more than
20 replicates to obtain accurate estimates.

6

Related work

The current state of NLG evaluation. Existing
approaches to NLG evaluation use a mix of qualityand diversity measures. Out of the 26 NLG papers
at ACL 2018, six perform only human evaluation,
fourteen measure human evaluation and a diversity
metric such as perplexity or n-gram diversity, andsix do not even evaluate using human judgments.
While perplexity and n-gram counts can inprinciple evaluate diversity, their practical implementations suffer from serious drawbacks. Whenhuman evaluation and perplexity are both evaluated, they are almost always done on separate
models – human evaluations are done on beamsearched output, while perplexity is computed on
the softmax outputs. This makes it appear as if themodels can simultaneously generate high quality
outputs while also being diverse, when in fact they
can only be one at a time based on whether theysample or run beam search.
On the other hand, n-gram diversity was proposed by (Li et al., 2016) to combat the generic utterance problem where models repeat phrases suchas ‘I don’t know’. Unfortunately, n-gram diversity is computed across contexts by counting thenumber of unique n-grams generated, and so does
not measure a model’s ability to generate multiple
valid utterances at any single context. In particular, a model which can only output a single utterance per context (e.g., via memorization or retrieval) can still have high n-gram diversity as longas the memorized sentences are unique.
Finally, all existing diversity measures are computed separately from human evaluation. This
results in two incomparable evaluation metrics,
which prevent us from reasoning about tradeoffs
between diversity and quality. In contrast, HUSE
allows us to make precise statements about thecost of diversity because it is a single metric which
decomposes into diversity and quality terms.
Related evaluations of diversity. The importance of diverse responses has previously acknowledged for summarization (Nenkova et al.,2007) and information retrieval (Clarke et al.,2008). Our work differs in considering a single
evaluation measure that captures quality and diversity applicable to any generation task.
Automated metrics based on n-gram overlap
such as BLEU, METEOR, ROUGE (Papineni
et al., 2002; Lavie and Denkowski, 2009; Lin andRey, 2004) work well for machine translation but
do not generalize well to domains with a diverse
spectrum of correct responses. While variants(Sun and Zhou, 2012; Galley et al., 2015; Shimaand Mitamura, 2011) have adapted such metrics
to high entropy generative environments, they arestill significantly inferior to the human judgments
they attempt to mimic.
Caccia et al. (2018) recently examined the diversity and quality tradeoffs for different language
model architectures on synthetic datasets. However, as their approach relies on measuring loglikelihoods under both the model and reference
distributions, it cannot be applied to real data
where pref is unavailable. Our main conceptual
contribution overcomes this by showing that HJ isan acceptable proxy for pref .
Sajjadi et al. (2018) also examines diversity andquality (which they call precision and recall) in
the context of generative image models. However, they rely on assuming that pref and pmodel
can be estimated accurately using the Fréchet
Inception Distance (FID) (Heusel et al., 2017).
HUSE avoids such assumptions and instead directly leverages the human judgments HJ, resulting in a simple and reliable metric more suitablefor use as a gold-standard.
Estimating optimal classification error. Evaluating a model by estimating its optimal classification error has been considered by several earlier
works (Olsson et al., 2018; Kannan and Vinyals,
2016; Li et al., 2017; Bruni and Fernandez, 2017;
Bowman et al., 2016). However, these methods have focused on classifying sentences directly
which is quite challenging to do reliably. In fact,
existing adversarial evaluation methods do not yet
reliably outperform human classification (Kannan
and Vinyals, 2016; Bruni and Fernandez, 2017).
We propose the use of both human evaluation andmodel probabilities as part of the adversarial evaluation framework, and demonstrate that the resulting classifier reliably outperforms humans andcaptures both the sample quality and diversity of amodel.

Distributional divergence estimation. Our
proposed evaluation metric is closely related tothe total variation distance which has been studied
extensively in the distribution testing literature.
It is known that total variation distance estimateshave pessimistic minimax estimation rates inhigh dimensions (Balakrishnan and Wasserman,2017). Our work overcomes this by utilizing
pmodel and an estimate of pref . Other approaches to
distributional testing include the maximum mean
discrepancy (MMD) and Wasserstein distances,which achieve better rates but at the cost of
relying strongly on a kernel and distance metricrespectively (Tolstikhin et al., 2016; Singh et al.,2018). Although such divergences are easierto estimate than the total-variation distance, theimplied convergence rates are still too slow to be
used as a replacement for human evaluation.

7

Discussion

In this paper, we demonstrate that the current gold
standard of human evaluation does not penalize
under-diverse models. To remedy this, we propose HUSE, a general purpose evaluation which
can be applied to any model for which we can calculate pmodel . HUSE is a upper bound to the optimal classification error of distinguishing reference
and model generated text, and never does worse
than human classification. HUSE leverages both
pmodel and HJ, ensuring that models which havehigh HUSE are both high-quality and diverse.
In many ways, our work can be seen as an extension to the classic Turing Test (Turing, 1950).
Instead of relying on just a human classifier, we
approximate the optimal classifier by adding additional information about the model through pmodel .
Finally, our work here focuses on natural language generation as a distribution matching task,
where the goal is pref ≈ pmodel . However, manysettings may have different goals. In machinetranslation (Bahdanau et al., 2015), for example,
a single high-quality output may suffice. Other
tasks may have external measures of model quality(task oriented dialogue), or there may be a desire
to generate with super human quality (story generation). In such cases, HUSE can be extended byconstraining the classifier f appropriately for thetask. Given that the current state of NLG has not
yet approached human level generation, we leavesuch investigations to future work.

References
D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neuralmachine translation by jointly learning to align andtranslate. In International Conference on Learning
Representations (ICLR).
S. Balakrishnan and L. Wasserman. 2017. Hypothesis
testing for high-dimensional multinomials: A selective review. arXiv preprint arXiv:1712.06120.
S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai,
R. Jozefowicz, and S. Bengio. 2016. Generating
sentences from a continuous space. In Computational Natural Language Learning (CoNLL), pages
10–21.
E. Bruni and R. Fernandez. 2017. Adversarial evaluation for open-domain dialogue generation. In Proceedings of the SIGDIAL 2017 Conference.
M. Caccia, L. Caccia, W. Fedus, H. Larochelle,
J. Pineau, and L. Charlin. 2018. Language gansfalling short. arXiv preprint arXiv:1811.02549.
A. Chaganty, S. Mussmann, and P. Liang. 2018. The
price of debiasing automatic metrics in natural language evaluation. In Association for Computational
Linguistics (ACL).
C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon.
2008. Novelty and diversity in information retrieval
evaluation. In ACM SIGIR.
M. Galley, C. Brockett, A. Sordoni, Y. Ji, M. Auli,
C. Quirk, M. Mitchell, J. Gao, and B. Dolan.
2015. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv
preprint arXiv:1506.06863.
J. Gehring, M. Auli, D. Grangier, D. Yarats, and
Y. N. Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122.
S. Gehrmann, Y. Deng, and A. M. Rush. 2018. Bottomup abstractive summarization. In Empirical Methods in Natural Language Processing (EMNLP).
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems(NeurIPS).
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler,
and S. Hochreiter. 2017. Gans trained by a two timescale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS).
R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer,
and Y. Wu. 2016. Exploring the limits of language
modeling. arXiv preprint arXiv:1602.02410.

A. Kannan and O. Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Workshop on
Adversarial Training.
G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M.
Rush. 2017.
Opennmt: Open-source toolkit
for neural machine translation. arXiv preprint
arXiv:1701.02810.
A. Lavie and M. Denkowski. 2009. The meteor metric for automatic evaluation of machine translation.
Machine Translation, 23.
J. Li, M. Galley, C. Brockett, J. Gao, and W. B.
Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Human Language Technology and North American Association
for Computational Linguistics (HLT/NAACL), pages
110–119.
J. Li, W. Monroe, T. Shi, A. Ritter, and D. Jurafsky.
2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547.
C. Lin and M. Rey. 2004. Looking for a few good metrics: ROUGE and its evaluation. In NTCIR Workshop.
T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona,
D. Ramanan, P. Doll’ar, and C. L. Zitnick. 2014.
Microsoft COCO: Common objects in context. In
European Conference on Computer Vision (ECCV),
pages 740–755.
C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Charlin, and J. Pineau. 2016. How NOT to evaluate yourdialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Empirical Methods in Natural Language
Processing (EMNLP).
R. Lowe, M. Noseworthy, I. V. Serban, N. AngelardGontier, Y. Bengio, and J. Pineau. 2017. Towards
an automatic turing test: Learning to evaluate dialogue responses. In Association for Computational
Linguistics (ACL).
N. Mostafazadeh, N. Chambers, X. He, D. Parikh,
D. Batra, L. Vanderwende, P. Kohli, and J. Allen.
2016. A corpus and cloze evaluation for deeper
understanding of commonsense stories. In North
American Association for Computational Linguistics (NAACL).
R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang,
et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv
preprint arXiv:1602.06023.
A. Nenkova, R. J. Passonneau, and K. McKeown. 2007.
The pyramid method: Incorporating human content
selection variation in summarization evaluation. In
ACM Transactions on Speech and Language Processing.

J. Novikova, O. Dušek, A. C. Curry, and V. Rieser.
2017. Why we need new evaluation metrics for
NLG. In Empirical Methods in Natural Language
Processing (EMNLP).
C. Olsson, S. Bhupatiraju, T. Brown, A. Odena, and
I. Goodfellow. 2018. Skill rating for generativemodels. arXiv preprint arXiv:1808.04888.
K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002.
BLEU: A method for automatic evaluation of machine translation. In Association for Computational
Linguistics (ACL).
M. Roemmele. 2016. Writing stories with help fromrecurrent neural networks. In Association for the
Advancement of Artificial Intelligence (AAAI).
M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. 2018. Assessing generativemodels via precision and recall. arXiv preprint
arXiv:1806.00035.
H. Shima and T. Mitamura. 2011. Diversity-aware
evaluation for paraphrase patterns.
In Empirical Methods in Natural Language Processing(EMNLP).
S. Singh, A. Uppal, B. Li, C. Li, M. Zaheer, and B. Poczos. 2018. Nonparametric density estimation underadversarial losses. In Advances in Neural Information Processing Systems (NeurIPS), pages 246–257.
A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji,
M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015.
A neural network approach to context-sensitive
generation of conversational responses. In North
American Association for Computational Linguistics (NAACL).
H. Sun and M. Zhou. 2012. Joint learning of a dual
SMT system for paraphrase generation. In Association for Computational Linguistics (ACL).
J. Surowiecki. 2004. The wisdom of crowds: Why the
many are smarter than the few and how collective
wisdom shapes business, economies, societies, andnations. Doubleday and Co.
L. Theis, A. van den Oord, and M. Bethge. 2015. A
note on the evaluation of generative models. arXiv
preprint arXiv:1511.01844.
I. Tolstikhin, B. K. Sriperumbudur, and B. Scholkopf.
2016. Minimax estimation of maximum mean discrepancy with radial kernels. In Advances in Neural
Information Processing Systems (NeurIPS), pages
1930–1938.
A. M. Turing. 1950. Computing machinery and intelligence. Mind, 49:433–460.
L. Ungar, B. Mellors, V. Satopää, J. Baron, P. Tetlock,
J. Ramos, and S. Swift. 2012. The good judgment
project: A large scale test of different methods of
combining expert predictions. In Association for the
Advancement of Artificial Intelligence (AAAI).

A

Appendix

A.1

Relationship between total variation
distance and optimal discriminator error

This is a standard result, replicated here for completeness:
Proposition 2. The total variation distance is related to the optimal discriminator error as follows: kpmodel − pref kTV = (1 − L∗ ).
Proof
def

def

Fix any x. Define ay = pref (y | x) anddef

by = pmodel (y | x). Let S = {y : ay < by }be the y where the pmodel assigns higher probadef P
bility than pref , and define A =y∈S ay and
def P
B =
y∈S by be the aggregated probabilities.
On S, the optimal discriminator should return z =0 (model). This is an error when z = 1, which
occurs with probability 12 A. Analogously, on thecomplement of S, the error probability (when z =0) is 12 (1 − B). The total contribution to L∗ is thus
A + (1 − B). The rest follows from algebra:
1
kpmodel − pref kTV = kpmodel − pmodel k1
2(10)1= [(B − A) + (1 − A) − (1 − B)](11)2= B − A = (1 − L∗ )
(12)

A.2

Amazon Mechanical Turk for humanjudgments

In order to show that HUSE can be reliably estimated even with simple crowdsourcing techniques, we used a single uniform task design
where we asked Amazon Mechanical Turk workers to rate the typicality of a sentence from 0-5.
We defined 0 as invalid (grammatically or factually incorrect) and 5 as ‘very typical’. HJ(x, y)
is defined as the average score that crowdworkers
assign to a response y given the context x. We
did not perform substantial filtering or qualification checks beyond HIT acceptance rate.
We observe that measuring many replicates issufficient to get low-variance estimates of HJ. For
easy classification tasks (such as story generation)
we require five to ten replicates, while for hardtasks such as summarization at least twenty replicates are needed (Section 5.4). Manual inspection suggests that up to 20% of the collected data

are low-quality but that this noise is uncorrelatedwith the sentence being rated and outweighed by alarger majority of honest and reasonably accurate
data.
A.3

Reddit Dataset

We use a subset of Reddit comments from 20062018 scraped from https://pushshift.io/. We construct a dictionary containing the 10000 most popular words and preprocess the dataset by removing deleted posts, out-of-vocabulary tokens, profanity, comments with less than 10 upvotes, andcomments with over 400 tokens.

Figure 6: Amazon Mechanical Turk survey design for eliciting HJ in the summarization task.