MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
Authors: Krishna Pillutla
Swabha Swayamdipta
Rowan Zellers
John Thickstun
Sean Welleck
Yejin Choi
Zaid Harchaoui
Venue: NeurIPS 2021 Oral (2021)
Keywords: Open-ended generation
neural text generation
evaluation
divergence frontier
human judgement

Abstract: As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce Mauve, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. Mauve scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that Mauve identifies known properties of generated text, scales naturally with model size, and correlates with human judgments, with fewer restrictions than existing distributional evaluation metrics.

MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
Krishna Pillutla
Swabha Swayamdipta" Rowan Zellers' John Thickstun
Sean Welleck
Yejin Choii.2 Zaid Harchaoui
Paul G. Allen School of Computer Science & Engineering, University of Washington Allen Institute for Artificial Intelligence
Department of Computer Science, Stanford University
Department of Statistics, University of Washington
Abstract
As major progress is made in open-ended text generation, measuring how close machine- geniiattd text is human language remains critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences quantized embedding space. Through
extensive empirical study three open-ended generation tasks, we find that MAUVE identifies known properties of generated text, scales naturally with model size, and correlates with human judgments, with fewer restrictions than existing distributional evaluation metrics.
Introduction
Recent large-scale text generation models show an ability to produce human-like text of remarkable quality and coherence in open-ended generation [45, 61, 6]. In this setting, text generation model forms distribution over natural language sequences, induced by an autoregressive neural sequence model (e.g., GPT-3 [6]) paired with decoding algorithm (e.g.. nucleus sampling [26]). Generating text amounts to sampling from this distribution, with the goal of obtaining samples that resemble those from the "true" distribution human- aomannwritten text.
To evaluate how close a generation model's distribution is to that of human-written text, we must consider two types of errors: (I) where the model assigns high probability to sequences which do not resemble human-written text, and, (II) where the model distribution does not cover the human distribution, i.c., it fails yield diverse samples. However, quantifying these aspects in a principled yet computationally tractable manner is challenging, as the text distributions are nigh-dimensional and discrete, accessed only through samples or expensive model evaluations [26, 58, 62].
We develop MAUVE, a comparison measure for open-ended text generation. The proposed measure efficient, interpretable, and practical for evaluating modern text generation models. It captures both types of errors (Figure 1) by building upon information divergence frontiers [49, 31, 16), so far underexplored in natural language processing. The key idea for making the proposed measure Dollback-Leiblee
computationally tractable, yet effective, is to reduce its measurement to computing divergences in guantized, fow-dimensional space after embedding samples from each distribution with an external language model. From an end-user's 's perspective, MAUVE has a simple interface: given neural text and human text, it yields a scalar measure of the gap between them.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Probability
Type Error:
Type just visited Utquagvik and Nuchallooogga In Alaska.
The time is
the time
the time
the time
Text
Figure 1: Left: MAUVE compares the machine text distribution Q to that of human text P by using the family of mixtures Ra >P+ X)Q for (0, 1). Right: Illustration of Type I errors, where Q produces degenerate, repetitive text which unlikely under P. and, Type II errors, where Q cannot produce plausible human text due to truncation heuristics [26]. MAUVE measures these errors softly, by using the mixture distribution Rx- Varying À in (0, 1) gives divergence curve and captures a spectrum of soft Type I and Type II errors. MAUVE summarizes the entire divergence curve in single scalar the area under this curve.
We summarize our contributions. First, introduce MAUVE, comparison measure between neural text and human text. Second, we empirically show that MAUVE is able to quantify known properties of generated text with respect to length, model size, and decoding more correctly and with fewer restrictions than existing distributional evaluation metrics. Third, we find through human evaluation that MAUVE better correlates with human quality judgements of text. Finally, we find that MAUVE can be highly robust to the choice of quantization, embeddings, and scaling. We open-source pip-installable Python package to compute MAUVE.
2 MAUVE
We begin by discussing the basics of open- ppen-ended text generation, and then introduce MAUVE for measuring the divergence between machine generated text and human text.
Open-ended Text Generation. A language model is estimate P(xx) of the probability distribution over sequences of text æ = (II,..... [ consisting of tokens x, belonging to a fixed vocabulary (e.g. characters, words). Prevailing neural autoregressive language models estimate the joint distribution P(x)) by modeling the conditional distribution P(x++1/<11d over the next token sequence. The open-ended text generation task asks us to output text &t+1:|m| in continuation of given context F1:t- Unlike targeted generation tasks like translation or summarization, there is no "correct" output; the main criteria for open-ended text generation are coherence, creativity, and fluency.
Given neural autoregressive language model P. right fashion, by sampling क P(-(æ1:)), ue42 P(- 2z+1), etc. In practice, this simple decoding algorithm often modified by adjusting the conditional distribution P(-[æ1:z) to promote more conservative outputs. The decoding algorithm and the language model taken together define distribution Q over text, which we call the model distribution. Common decoding algorithms include temperature rescaling [1] and truncation [18, 26]. Note that truncation methods in particular create sparsity Q, which leads to degeneracy of some measures including test-set perplexity.
generate open-ended text in serial, left-to-
Sources of Error in Text Generation. Our goal in this work is to measure the gap between the model distribution Q and the target distribution of human text. As highlighted in Figure this gap arises from two sources of error:
(Type 1) Q places high mass text which is unlikely under P. (Type II) Q cannot generate text which is plausible under P.
The Type errors are false positives, including the common failure case where a model generates text with semantic repetitions [15, 26, 59] that highly unlikely to be written by humans. The Type II
Available from https: //g::///iii http//gttthuttteeeee/eee com/ htt.aa....o....ss/aaasaeeeeee http://risamma/m/am.e.... See Appendix B for example of the mauve package in action.
'Let text æ with Pxx) > 0 be the positive class and P(æ) ~ 0 be the negative class. If Q(x) > 0 for some negative æ, then the model incorrectly considers it positive, it a false positive.

---

GPT-2 Size for Ancestral Samp
Grooerr Size for Nucleas Sampling
Decoding Alp. for GPT-2 large
Decodinn Alg. for Grever base
eapt- -o0---
expic coppr
eapt- Poc-coopp.nn
Nuclous
Sampling
Greedy
xI
mega
Nuciaus Sampling
84 14
23
07
axp(-cDIQ|RR)
DICOALD
sxp(=cD(0jR))
D(Q)RR))
Divergence curves for different models (GPT-2 [45], Grover [61]) and decoding algorithms (greedy decoding, ancestral and nucleus sampling). MAUVE is computed as the area of the shaded region, and larger values of MAUVE indicate that is closer P. In general, MAUVE indicates that generations from larger models and nucleus sampling are closer human text. Rightmost: Nucleus sampling has slightly smaller Type error than ancestral sampling but a higher Type II error, indicating that ancestral sampling with Grover basc produces more degenerate text while nucleus sampling does not effectively cover the human text distribution.
Figure 2:
errors are false negatives, which can occur, for instance, because some pieces of plausible human text cannot be generated by gruncation-based decoding algorithms such as nucleus sampling [26]. The gap between P and Q is small only if both of these errors are small.
Quantifying the Errors. We formalize the Type and II emross with the Kullback- (KL) divergences (L(()) P) and (L(PIQ), respectively. The divergence (Q) penalizes Q if there exists text such that (xx) is large but P(æ) is small, so it quantifies the Type error. Likewise, KL(PIQ) quantifies the Type error.
Unfortunately, one or both of the KL. divergences (IQ( and [I() are infinite if the supports of and Q are not identical, which often the case in open-ended generation. This makes the KL divergence itself unsuitable
evaluation metric. We overcome this issue by softly measuring the two errors using the mixture distribution Rx = XP - (1 - X)Q for some À E (0, 1). In particular, we define the (soft) Type I error at level À as (L(Q))xx and the (soft) Type II error as KL(P)Rx).
Summarizing the Errors with a Divergence Curve. Since the mixture weight À was arbitrary, we consider a family of Type and II error values by varying À between 0 and 1, in the same spirit as information divergence frontiers [49, 16]. This yields divergence curve,
Q(),Q) = (
(xp(e)KL(Pppppp)Xeee (P)))) : RA = XP + (1 N)Q, AE (0. 1}.
where e is a hyperparameter for scaling. The divergence curve formalizes and encodes informa- tion about the trade-off between Type and II errors.3 Figure 2 illustrates the divergence curves for different models and decoding algorithms.
Our proposed measure, MAUVE P, Q), is the area under the divergence curve C(P, Q). It provides a scalar summary of the trade-off between Type and Type II errors. MAUVE(P, Q) lies in (0, 1]. with a larger value meaning that Q is closer to P. Further, MAUVE(P, Q) = 1 if and only if Q = P. The area under the curve a common summary of trade-off curves in machine learning [13, 11, 12, 19].
Connections to Common Divergences. The divergence curve encodes more information than the KL divergence (L(P)Q), which be obtained from the second coordinate of the curve C(P,Q) Q) as X - O, and the reverse KL divergence (L())P) which can be obtained from the first coordinate of the curve C(P, JS(P, Q) (KL(P)R1/2) + KL(Q R 22) )
X 1. Further, the Jensen-Shannon (JS) divergence /2 2, can be obtained from the two coordinates of C(P, Q) at A = 1/2. MAUVE summarizes all of the divergence curve C(P,Q).
Computing MAUVE for Open-Ended Text Generation. Each point on the divergence curve C(P,Q) consists of a coordinate
P(x:) Rx(x)*
KL(P Σ P(x) log
and a similarly defined coordinate ( We cannot compute the summation as written in Eq. (2), as we do not know the ground ground-truth probabilities P(.) and the support of a typical model
'More generally, the divergence curve C(P, Q) encodes the Pareto frontier of (KL(PIR), (L)R)) for all distributions R, not just mixtures of the form Hx. We prove this in Appendix A.
3

---

6
6
4
3 8
3
10
11
8
10
11
Illustration of the quantization. Left: A continuous two-dimensional distribution P. Right: A partitioning of the Euclidean plane IR? and the corresponding quantized distribution
Figure 3:
distribution is prohibitively large, since it is the space of all sequences of tokens. As a result of these two issues, MAUVE cannot be tractably computed in closed form.
We employ Monte Carlo estimator using samples a; P and a² Q to overcome the fact that ground-truth probabilities P(-) are unknown. We circumvent the intractable support size by computing MAUVE in quantized embedding space that is sensitive important features of text.
The overall estimation procedure as follows. First, we sample human text æ; P and machine text ay- Q. We then embed each text sequence using an external language model M (e.g.. GPT-2 [45]) to obtain embeddings (x,)) and
Each embedding is vector (x)) R4. Next, we jointly quantize the embedded samples (c.g. with k-means [36]), and count the cluster assignments to form histograms, giving dinennnnsional discrete distributions that approximate each high-dimensional text distribution. In particular, the distribution P of human text is approximated by the discrete distribution P of support size k; which is defined as,
N P()
Σ ((((z.) = N
(3)
where ((x) E {1,.... k} returns the cluster id of æ. The model distribution Q is approximated as Q similarly. Here. P and Q can be interpreted piecewise constant approximations of P and Q, similar to histogram; see Figure 3 for an illustration. Computing the divergence curve is now tractable, each coordinate is KL divergence between the k-element discrete distributions.
To recap, our proposed measure MAUVE( (P. Q) the area under this divergence curve, providing summary of all Type and Type II errors through an efficient approximation designed for text generation. Next, we discuss how MAUVE compares to prior comparison measures for text ($3), then present empirical results with MAUVE ($4).
3 Related Work
Divergence Measures for Text. Prior measures of similarity'divergence between machine text and human text come in three broad categories: (a) ofference-based (b) cattistics-basede and (c) language modeling. Table summarizes the latter two categories, and contrasts them with MAUVE.
Reference-bused measures evaluate generated text with respect (small set of) reference text sample(s), rather than comparing full sequence distributions. These include classical metrics for n-gram matching [44, 32, 2]. which are designed to capture similarities in the surface form of the generated text and the human references, making them fundamentally ill-suited for open-ended generation. Moreover, it has been recently shown in [42] show that these classical metrics only weakly agree with human judgments.
More recent reference-basee metrics are capable of comparisons in high dimensional space [53, 63, 51, 9], thereby capturing distributional semantics beyond superficial n-gram statisties. For instance, Moverscore [64] relies on the Word Mover's distance [30], and is an instance of an optimal transportation distance [57]. computes the minimum cost of transforming the generated text the reference text, taking into account Euclidean distance between vector representations of n-grams,

---

Type
Metric
Measures
Approximates
Zipf Coefficient [26]
Unigram rank- men-frrquenccy statistics
Statistics
Belf-BLEU [65]
N-gram diversity
Generation Perplexity
Generation quality via external model It
Eg log R(ar) Brllog R(x)]]
[18]
(a single point inside C(P. Q))
Perplexity
Test perplexity
Epllog Q(x:) [r[Q(=)]
Language
e-perplexity [39]
Perplexity w/ Laplace smoothing
Modeling
Sparsemax Score [39]
LM quality (sparsemax loss (38])
[p(Q(=)
Token JS-Div. [39]
LM quality (IS divergence)
[p(((x))
Quality & diversity via the divergence curve
Divergence MAUVE (this work) Curve
C(P. Q) all
Table 1: Summary of automatic distributional metries for evaluating open-ended text generation. MAUVE provides a summary of all points along the divergence curve, rather than single point. The summary is based comparisons in a joint embedding space, rather than statistic computed independently on each distribution. Q informally refers to quantity related to Q.
well their document frequencies. The paradigm of reference-based measures is useful for targeted generation tasks such as translation and summarization where matching set of references is paramount. is. however, unsuitable for open-ended generation where there typically are several plausible continuations for each context and creative generations are desirable.
Statistics- based measures compare the model distribution Q with respect to the human distribution P on the basis of some statistic T(P) and TQ)). Property-specific statistics such as the amount of repetition [26, 59], verifiability [40]. or termination [58] are orthogonal to MAUVE, which provides a summary of the overall gap between P and Q rather than focusing on an individual property. Another statistic is the generation perplexity [18, 26), which compares the perplexity of the model text a ~ Q with that of human text x' P under an external model R. By virtue of T(-) being scalar, generation perplexity cannot trade-off the Type and Type II errors like MAUVE. In fact, we show in Appendix A that the generation perplexity can be derived from a single point enclosed between the divergence curve and the axes.
Language modeling metrics calculate how (un)likely human text æ ~ P under the model distribu- tion Q. for instance, using the probability Q(x).. These metrics are related to single point the divergence curve, rather than a full summary. Examples include the perplexity of the test set (which sample from P) under the model Q and its generalizations to handle sparse distributions [39]. Unlike MAUVE, these metrics never see model text samples x'' ~ Q, so they cannot account for how likely the model text under the human distribution P. Moreover, they cannot be used for decoding algorithms such as beam search which do not define a token-level distribution.
Automatic metrics have been proposed for specific domains such generation of dialogues [55], stories [21], and others [43]. They capture task-specific properties; see the surveys [8, 48). In contrast, MAUVE compares machine and human text in domain-agnosti manner. Other related work has proposed metrics that rely on multiple samples for approaches to compare the distribution of statistics in machine translation [17].
evaluation [7], and Bayesian
Non-automatic Metrics. HUSE [24] aims to combine human judgements of Type errors with Type II enrors measured using perplexity under Q. Due to the costs of human evaluation, we consider HUSE, as well other metrics requiring human evaluation, such as single- sinle-pair pair evaluation, complementary to MAUVE, which is automatic comparison measure. As separate technical caveat, it is unclear how to use HUSE for sparse Q that assigns zero probability to a subset of text, which is the case with state-of-the-an art decoding algorithms [26, 39].
Evaluation of Generative Models. Evaluation of generative models is an active area of research in computer vision, where generative adversarial networks [20] are commonly used. However, metrics such as Inception Score [50] are based on large-scale supervised classification tasks, and thus inappropriate for text generation. The Fréchet Distance [25, 52] and its unbiased counterpart, the Kernel Inception Distance [5] are both used for evaluating generative models, but unlike MAUVE, do not take into account trade-off between different kinds of errors between the learned and
5

---

Prompt Length
Max. Generation
Number of Generations
Task Domain
Model
Finetuning
Dataset
Length
Web
GPT-2 (all sizes)
Pretrained
Webtext
35 tokens
1024 tokens
5000
News
Grover (all sizes)
Pretrained
RealNews
varying
1024 tokens
5000
Stories
GPT-2 medium
Finetuned
WoitinPrompls
50 tokens
512 tokens
5000
Table 2: Dataset task summary. Note that 1024 tokens correspond to ~ 750 words on average.
reference distribution. Sajjadi et al. [49] and Kynkäänniemi al. [31] both proposed metrics based on precision-recal curves. Djolonga et al. [16] proposed information divergence frontiers as a unified framework emcompassing both these works as special cases. MAUVE extends the above line of work, and is operationalized for open-ended text generation, applicable for data generated by large-scale neural language models. Complementary to this work, Liu et al. [33] study the theory of information divergence frontiers, proving non-asymptotic bounds on the estimation and quantization error.
4 Experiments
We perform three sets of experiments validate MAUVE. Our first set of experiments ($4.1) examine how known properties of generated text with respect to generation length, decoding algorithm, and model size can be identified and quantified by MAUVE. Next, in $4.2 we demonstrate that MAUVE robust to various embedding strategies, quantization algorithms, and hyperparameter settings. Finally, in $4.3 we find that MAUVE correlates with human judgments. The code as well as the scripts to reproduce the experiments are available online.*
Tasks. We consider open-ended text generation using text completion task [26, 59] in three domains: web text, news and stories. Each domain consists of sequence dataset split into (context, continuation) pairs. Given a context #ick- the task is to generate a continuation & Q(- Ø1:k), forming a completion. Each ground-truth completion Z1.r is considered a sample from the true distribution P. while the completion (I1:k) #k+1:T) is considered a sample from Q. The datasets, context and completion lengths, and number of completions used for each domain are shown in Table
Models. As the language model P(.), we use GPT-2, a large-scale transformer [56] pretrained on the web text dataset (see [45]), that is representative of state-of-the-art autoregressive language models. As the embedding model (-) we use GPT-2 Large, and compare others in $4.2.
Decoding Algorithms. We consider three common decoding algorithms: samples directly from the language model's per-step distributions, Ie P(a, x1.)). greedy decoding which selects the most likely next token, 2, arg max P(x (1:)) as well nucleus sampling [26) which samples from top-p truncated per-step distributions, Ia Pomeppat araa) which is defined as
ancestral sampling which
[ume,p(a æ1s), if i E Vp. (0.
P poe x1s) x
else.
Here, the top-p vocabulary Vi, is the smallest set V such that Evev P(z x.d) 2 p.
We also consider an adversarial sampling procedure, designed generate low-quality text that nevertheless matches the perplexity of human text. Adversarial perplexity sampling proceeds in two phases: (1) we generate the first 15% of tokens in a sequence uniformly at random from the vocabulary, and (2) we generate the remaining tokens greedily to make the running perplexity of the generated sequence as close as possible to the perplexity of human text.
4.1 Quantifying Properties of Generated Text
To study MAUVE'S effectiveness as measure for comparing text distributions, we first examine how MAUVE quantifies known properties of generated text: good measure should meet expected behavior that is known from existing research on each property. Specifically, we investigate how MAUVE behaves under changes in generation length, decoding algorithm, and model size.
'nttps://githann http:/ttw..ccom/rsssaaa//////// experimeneee..........e

---

Metrics versus Max. Text Length for Nucleus Sampling
MAUVE (T)
Fréchet Distance (4)
Gen. PPL. diff. ( (4)
Sparsemax score (1)
Fréchet Distance
12.
MAUVE
Sparsemax
0.92
12.0
Gen.
0.67
0.90
11.8
0.03
11.6
200
420
500
1000
200
600
600
3000
250
500
750
1000
200
400
600
300
1000
Max. Length
Max. Length
Max. Length
Max. Length
small
medium
large
Figure 4: Generation quality versus maximum generation length according MAUVE and three alternative measures (web text, GPT-2). MAUVE the only comparison measure which identifies that generation quality decreases monotonically with increasing text length. The shaded area shows one standard deviation over generations from 5 random seeds.
MAUVE quantifies quality differences due to generation length. Although large transformer- based models can generate remarkably fluent text, it has been observed that the quality of generation deteriorates with text length: as the generation gets longer, the model starts to wander, switching to unrelated topics and becoming incoherent [46]. As result, an effective measure should indicate lower quality (c.g. lower MAUVE) as generation length increases.
Figure 4 shows MAUVE the generation length increases, along with three alternative metrics: generation perplexity, sparsemax score, and Fréchet distance [25, 52]. MAUVE reflects the desired behavior, showing a decrease in quality (lower MAUVE) as generation length grows, with the trend consistent across model sizes. The other three metrics, however, show less favorable trends. Fréchet distance indicates improving quality as the length increases, while generation perplexity shows non-monotonic quality trends for the small and large models. Finally, language modeling metrics such as the sparsemax score [39] remain constant, since they do not depend on the samples generated.
MAUVE identifies quality differences between decoding algorithms. Recent work has identified two clear trends in open-ended text generation with standard autoregressive models: (1) using greedy decoding results in repetitive, degenerate text [26, 59, 58]; (2) nucleus sampling (and related truncated sampling methods) yields higher quality text than ancestral sampling [18, 26). An effective measure should thus indicate the quality relationship greedy < ancestral < nucleus.
Table 3 summarizes MAUVE's quality measures of greedy decoding, ancestral sampling, and nucleus sampling, along with alternative automated metrics and a human quality score. MAUVE correctly identifies the expected quality relationship. assigning the lowest quality to greedy decoding (.016) followed by ancestral sampling (.882), and the highest quality to nucleus sampling (.940). Other commonly-used metrics fail to identify this relationship: generation perplexity rates the highly degenerate greedy decoded text as better than ancestral sampling (11.324 vs. 19.284), while the language-modell metries (SP, JS, e-PPL) rate nucleus-decoded text equal to worse than greedy decoding or ancestral sampling. Further, as we show in Appendix D, MAUVE rightly identifies degeneracy of beam search, thus quantifying the qualitative observations of Holtzman al. [26). Finally, generation perplexity falls victim to the adversarial decoder (Adv.), unlike MAUVE.
MAUVE quantifies quality differences due to model size. Scaling the model size has been key driver of recent advances in NLP, with larger models leading to better language modeling and higher quality generations in open-ended settings [45, 6]. An effective metric should capture the relationship between model size and generation quality, which we verify with human quality scores.
Table 4 shows MAUVE'S quality measures as the model size increases, along with alternatives and human quality scores. MAUVE increases as model size increases, agreeing with the human quality measure and the expectation that larger models should have higher quality generations. The widely- used generation perplexity, however, incorrectly rates the large model's is text as the best. Although the language modeling metrics (SP, JS, and s-PPL.) capture the size-quality relationship, they are constant with respect to length (Figure 4), and did not correctly quantify decoding algorithm quality (Table 3).
'In general this relationship depends on nucleus hyperparameter p and task. Here, we follow the same settings as Holtzman et al. [26], and additionally include a human assessed measure of quality. 'The results are consistent across model sizes and random seeds (see Appendix D).
7

---

Adv. Greedy Sampling Nucleus
Small
Medium
Large
XL
Gen. PPL(L) 0.05
11.3
19.3
1.54
Gen. PPL(()
11.2
8.5
0.9
1.5
Zipf(L)
0.03
0.02
0.02
0.01
Zipf(()
0.06
0.00
0.02
0.01
Self-BLEU()) 0.07
0.03
0.02
0.03
Self-BLEU(I)
0.05
0.02
0.03
0.03
SP(()
0.50
0.69
0.69
SP(()
0.65
0.67
0.68
0.69
JS(.)
0.35
0.37
0.36
JS(()
0.41
0.39
0.37
0.36
s-PPI.())
497
11.4
13.7
5- s-PPI.())
25.9
18.8
14.9
13.7
MAUVE (1) 0.06
0.02
0.88
0.94
MAUVE (1)
0.878
0.915
0.936
0.940
Human(t)
9.0
15.7
Human(1) 15.9
-3.4
12.6
15.7
Table 3: Generation quality w.r.t different decoding algorithms (web text, GPT-2 xI) under various met- rics, and humans. MAUVE correctly captures the re- lationship greedy < ancestral nucleus, and rates the adversarial decoder's text as low quality. Re- sults are consistent across model sizes and random seeds. Boldfaced/highhighhe entries denote the best decoding algorithm under each metric.
Table 4: Generation quality w.r.t different model sizes (web text, nucleus sampling) under various met- rics, as well as human evaluators. MAUVE captures the relationship between model size and generation quality, agreeing with human evaluated quality, Re- sults are consistent across random seeds and decod- ing algorithms, Boldfaced/highligheed entries denote the hest model size under each metric.
Table in Appendix D shows additional results with ancestral sampling- In this case, human evaluators rated generations from the small model as better than those from the medium model. Interestingly, MAUVE also identified this relationship, agreeing with the human ratings, in contrast to the other automatic metrics we surveyed.
Summary. MAUVE identifies properties of generated text that a good measure should capture, related to length, decoding algorithm, and model size. In contrast, commonly used language modeling and statistical measures did not capture all of these properties. Unlike these alternatives, which capture a single statistic or relate to single point on the divergence curve. MAUVE's summary measure incorporates type errors that quantify the degenerate text produced by greedy decoding (recall Figure 1), while capturing distribution-leve information that describes quality changes from generation length, model size, and the nuanced distinction between ancestral and nucleus sampling.
4.2 Approximations in MAUVE
MAUVE summarizes the divergence between two text distributions with approximation that relies on two components: an embedding model (x) and a quantization algorithm AA ($2, Eq. (3)). We study the effects of these two components.
MAUVE works with alternative embedding models. Figure (left) shows that MAUVE with features from RoBERTa- large [34] gives qualitatively similar trends across model size and decoding as MAUVE with features from GPT-2 large. Quantitatively, the Spearman rank correlation between them across all model and decoders is 0.993. We observe that RoBERTa penalizes smaller models more than GPT-2 but rates greedy decoding higher. We leave further study of inductive biases the different embedding models to future work.
MAUVE robust quantization. We compare different three different quantization algorithms:
(a) h- MeMan:: We cluster the hidden representations using k-means, and represent them by their cluster membership get a discrete distribution with size equal to the number of clusters.
(b) Deep Residual Mixture Models (DRMM): As generalization of k-means, we train a deep generative model known DRMM [22]. We convert the soft clustering returned by DRMM into a hard clustering by assigning each point to its most likely cluster, and quantize the data using the cluster membership. We use DRMM with layers and 10 components per layer for total of 10y clusters, and train it for 20 epochs.
(c) Lattice Quantization: We learn a 4-dimensional feature representation of the vectors (ar) using a deep network which maintains the neighborhood structure of the data while encouraging the features be uniformly distributed on the unit sphere [47]. We quantize the data on uniform lattice into 744 bins.
8

---

MAUNE for Nuceus Sampling
MAUVE for GFT
MALVE for Nucleus Sampling
MAUVE for GPT-2 xi
if
MALVEE
633
Features
Features
Scaling constant
Scaling constant
small
Sempling
Nucleus
small
large
Sampling
Nucleus
medium
Greedy
medium
Greedy
Figure 5: Left: MAUVE computed using GPT-2 (default) and RoBERTa [34] embeddings, across model sizes and decoding algorithms; see Table in the Appendix for further results. The Spearman rank correlation between the two is 0.993 across model sizes and decoding algorithms. Right: Fifecct of the scaling constant e on MAUVE. Choice of e does not affect the relative order of the curves but only the numerical value. We use 5 get interpretable values with both nucleus and greedy decoding.
We compare different choices of the quantization to k-means with k = 500, which is our default. The Spearman rank correlation between MAUVE computed with k-means for k ranging from 100 to 5000 correlates nearly perfectly with that 0.99 1.00. Likewise, MAUVE computed with DRMM lattice quantization has a near near-perfect Spearman correlation of at least 0.99 with k-means. While the actual numerical value of MAUVE could vary with the quantization algorithm, these results show that the rankings induced by various variants of MAUVE are nearly identical.
500. In particular, the Spearman correlation is exactly
Practical recommendation for scaling parameter. Figure 5 (right) shows the effects of adjusting the scaling parameter c, which does not affect the relative order of the divergence curve, but adjusts the numerical value returned by MAUVE. As a practical recommendation, we found e = 5 to yield interpretable values.
4.3 Correlation with Human Judgments
An effective metric should yield judgments that correlate highly with human judgments, assuming that human evaluators represent gold-standard." We evaluate how MAUVR's quality judgments correlate with human quality judgments. In our study, quality judgment means choosing a particular (model, decoder) setting based on the resultant generations.
Evaluation Protocol. To obtain human judgments, we employ a pairwise setup: at each round, an annotator receives a context and continuations from two different (model, decoder) settings, and selects the continuation they found more natural using 5-point Likert scale. Our interface for collecting annotations is shown Figure 9 of Appendix E, which also includes further details and additional results.
We collect these annotations for web text generation with 8 different (model, decoder) settings plus a ninth setting for human-written continuations. Each setting is a GPT-2 model size paired with either ancestral nucleus sampling. This gives us a total of 36 pairs of settings. Given the known difficulties with human evaluation of longer texts [28], we use a maximum completion length of 256 tokens. We obtain 90 preference ratings for each pair of settings, coming from total of 214 crowd-workers from the Amazon Mechanical Turk platform. The evaluators were paid USD 0.40 per evaluation based on an estimated wage of USD 16 per hour.
We convert these pairwise preferences to a ranking by fitting Bradley-Tere Terry model [37], parametric model used to predict the outcome of head-to-heat comparison. In particular, we obtain a score w; for each setting so that the log odds of humans preferring setting i to setting j in a head-to-head comparison is given by the difference ut; wy. For given comparison measure, Spearman rank correlation between the comparison measure and the fitted Bradley-Terry coefficients w; for each of the (model, decoder) settings. The end result is a correlation score in [-1, 1], with higher values meaning that quality judgments using the comparison measure correlate with quality judgments made by human evaluators.
compute the
"Concurent work has shown that human evaluation might not always be consistent [10, 29]; however human judgments continue to be the gold standard for evaluating open-ende ended text generation.

---

Metric
Task
Gen. PPL
Zipf Coet.
REP
Distinct-4
Self-BLEU
MAUVE
Human-likehi Interesting/BIT Sensible/BT % Disc. Acc. % Disc. Acc.
Weh text
0.810
0.833
-0.167
0.738
0.595
0.952
Web text
0.643
0,524
-0.143
0.524
0.405
0.810
Web text
0.738
0,690
-0.071
0.595
0.524
0.857
News
0.468
0,595
0.792
0.653
0.516
0.956
Stories
0.643
0.643
0.250
0.750
0.857
0.893
Table 5: Correlation of various similarity measures with human judgments when available, and the accuracy of a trained discriminator otherwise. "BT" denotes the Bradley-Ters Terry score for a pairwise human evaluation ($ 4.3). Boldfaced/highlighted numbers indicate highest correlation in each row. We observe that MAUVE has the highest correlation with human evaluation and discriminator accuracy.
MAUVE correlates with human judgments. Table 5 shows the correlation between human judg- ments and five automatic evaluation metrics obtained using our evaluation protocol on the web text domain. MAUVE correlates highly with human judgments of how human- like (0.952), interesting (0.810), and sensible (0.857) the machine text is. MAUVE's correlations with human judgments are substantially higher than those for the other automated measures; for instance, the commonly-use generation perplexity has correlations that are 0.12 to 0.17 lower than MAUVE'S. The results suggest that MAUVE may act as an effective, automatic surrogate for costly human judgments.
MAUVE correlates with learned discriminators. We also measure the quality of generations by how well trained model (a discriminator) can distinguish between real and generated text [35]. We report the test accuracy of binary classifier trained discriminate between machine and human text; a lower discrimination accuracy implies that the generation is harder to distinguish from human text. We report the accuracy of Grover mega as the discriminator for the news generations as produced the highest discrimination accuracy [61] while we use GPT-2 large for the story domain. As seen in Table 5, MAUVE correlates the highest with the discrimination accuracy (0.96 for news and 0.89 for stories) among all comparison measures. Computing the discrimination accuracy for each (model, decoder) pair requires fine-tuning a separate model, which is particularly expensive for large models such Grover-mega. MAUVE, the other hand, does not require any training.
5 Conclusion
We presented MAUVE, an automatic measure of the gap between neural text and human text for open-ended text generation. MAUVE measures the area under a divergence curve, formalizing and summarizing a spectrum of eгrоrs that capture phenomena present in machine and human- gennaan-eeerteedd text. MAUVE correlated with human judgments and identified quality differences due generation length, decoding algorithm, and model size, which prior metrics struggle to capture. Automated metrics have driven advances in computer vision and many other machine learning domains. MAUVE's principled foundation and strong empirical performance offers a similar path forward for open-ended text generation systems. Extensions of MAUVE to closed-ended tasks, such as summarization and translation, where generations must be compared to fixed set of gold-standard references, are promising directions for future work.
Broader Impacts Statement MAUVE rewards model text which resembles human authored text. However, we acknowledge the risks of rewarding systems that try to mimic humans [4], which is the ultimate goal of open-ended text generation. While our research is important for developing better language generators, we also encourage the community to pay attention to the development of technology that can reliably distinguish between human and machine text. We leave the extension of our method towards building such systems to future work.
Acknowledgments Part of this work was done while Zaid Harchaoui was visiting the Simons Institute for the Theory of Computing. and while John Thickstun was at the University Washing- ton. This work was supported by NSF DMS-21 34012, NSF CCF-2019844, NSF DMS-2023 166. the DARPA MCS program through NIWC Pacific (N66001-19-2-4031) the CIFAR "Learning in Machines & Brains" program, a Qualcomm Innovation Fellowship, and faculty research awards.
DMS-2134012,
10

---

References
[1] D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. A learning algorithm for Boltzmann machines. Cognitive science, 91):1477-169, 1985.
[2] S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 63-72, 2005.
[3] A. Belz, S. Mille, and D. M. Howcroft. Disentangling the Properties of Human Evaluation Meth- ods: A Classification System to Support Comparability, Veta-Evaluation and Reproducibility Testing. In Proc. of INLG, pages 183-194, 2020.
[4] E. M. Bender, T. Gebru, A. McMillan- -Mjorr, and S. Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proc. of FAccT, 2021.
[5] M. Binkkwski,, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD GANs. In Proc. of ICLR, 2018.
[6] B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan,
P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child,
A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Gray,
B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners. In Proc. of NeurlPS, 2020.
[7] M. Caccia, L. Caccia, W. Fedus, H. Larochelle, J. Pineau, and L. Charlin. Language GANs Falling Short. In Proc. of ICLR, 2020.
[8] A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of Text Generation: A Survey, arXiv Preprint, 2020.
[9] E. Clark, A. Celikyilmaz, and N. A. Smith. Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts. In Proc. of ACL, 2019.
[10] E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith. All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. In Proc. of ACL, 2021.
[11] S. Clémençon and N. Vayatis. Nonparametric estimation of the precision- curve. In Proc. of ICMI, pages 185-192, 2009.
[12] S. Clémençon and N. Vayatis. Overlaying classifiers: practical approach to optimal scoring. Constructive Approximation, 32:619-648, 2010.
[13] C. Cortes and M. Mohri. Confidence intervals for the area under the ROC curve. In Proc. of NeurlPS, volume 17, 2005.
[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In Proc. of NAACL, pages 4171-4186, 2019.
[15] E. Dinan, V. Logacheva, V. Malykh, A. Miller, K. Shuster, J. Urbanek, D. Kiela, A. Szlam, I. Ser- ban, R. Lowe, S. Prabhumoye, W. Black, A. Rudnicky, J. Williams, J. Pincau, M. Burtsev, and J. Weston. The Second Conversational Intelligence Challenge (ConvA12), 2019.
[16] J. Djolonga, M. Lucic, M. Cuturi, O. Bachem, O. Bousquet, and S. Gelly. Precision-Recal Curves Using Information Divergence Frontiers. In Proc. of AISTATS, pages 2550-2559, 2020.
[17] B. Eikema and W. Aziz. Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation. In Proc. of CoLING, 2020.
[18] A. Fan, M. Lewis, and Y. N. Dauphin. Hierarchical Neural Story Generation. In Proc. of ACL, pages 889-898 889-898,20 2018
[19] Flach. Machine Learning: The Art and Science of Algorithms That Make Sense of Data. Cambridge University Press, 2012.
[20] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Networks. In Proc. of NeurIPS, 2014.
[21] Guan and M. Huang. UNION:
An Unreferenced Metric for Evaluating Open-ended Story Generation. In Proc. of EMNLP, pages 9157-9166, 2020.
[22] P. Hämäläinen and A. Solin. Deep Residual Mixture Models. arXiv preprint, 2020.

---

[23] T. S. Han and K. Kobayashi. Mathematics of Information and Coding, volume 203, American Mathematical Soc., 2007.
[24] T. Hashimoto, H. Zhang, and Liang. Unifying human and statistical evaluation for natural language generation. In Proc. of NAACL, pages 1689-1701, 2019.
[25] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two -Smel Update Rule Converge to a Local Nash Equilibrium. In Proc. of NeurlPS, page 6629-6640, 2017.
[26] A. Holtzman, Buys, M. Forbes, and Y. Choi. The Curious Case of Neural Text Degeneration. In Proc. of ICLR, 2020.
[27] D. R. Hunter. MM algorithms for generalized Bradley-Tery Terry models. The Annals of Statistics, 32(1):384-406, 2004.
[28] D. Ippolito, D. Duckworth, C. Callison-Burre and D. Eck. Automatic Detection of Generated Text is Fasiest when Humans are Fooled. In Proc. of ACI, pages 1808-1822, July 2020.
[29] M. Karpinska, N. Akoury, and M. lyyer. The Perils of Using Mechanical Turk to Evaluate Open-Ender Text Generation. In Proc. of EMNLP, 2021.
[30] M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger. From Word Embeddings to Document Distances. In Proc. of ICML, pages 957-966. PMLR. 2015.
[31] T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved Precision and Recall Metric for Assessing Generative Models. In Proc. of NeurIPS, 2019.
[32] C.-Y. ROUGE: Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74-81, 2004.
[33] L. Liu, K. Pillutla, S. Welleck, S. Oh, Y. Choi, and Z. Harchaoui. Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals. In NeurlPS, 2021.
[34] Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and
V. Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv Preprint, 2019.
[35] D. Lopez-Paz and M. Oquab. Revisiting Classifier Two-Sample Tests. In Proc. of ICIR, 2017.
[36] C. Manning and Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 2001. ISBN 978-0-262-18 13360-9-
[37] J. I. Marden. Analyzing and modeling rank data, volume 64 of Monographs on Statistics and Applied Probability. Chapman & Hall, London, 1995. ISBN 0-412-99521-2.
[38] A. Martins and R. Astudillo. From Softmax to Sparsemax: A Sparse model of Attention and Multi-label Classification. In Proc. of ICMI., pages 1614-1623. PMIRR 2016.
[39] H. Martins, Z. Marinho, and A. T. Martins. Sparse Text Generation. In Proc. EMNLP, pages 4252-4273, 2020.
[40] L. Massarelli, F. Petroni, A. Piktus, M. Ott, T. Rocktäschel, V. Plachouras, F. Silvestri, and
S. Riedel. How Decoding Strategies Affect the Verifiability of Generated Text. arXiv preprint arXiv: 19/ 1.03587, 2019.
[41] K. Miettinen. Nonlinear Mulliobjectiive Optimization, volume 12. Springer Science & Business Media, 2012.
[42] J. Novikova, O. Dušck, A. Cercas Curry, and V. Rieser. Why We Need New Evaluation Metrics for NLG. In Proc. of EMNLP, 2017.
[43] Opitz and A. Frank. Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR. In Proc. of EACI. pages 1504-1518, 2021.
[44] K. Papineni, S. Roukas, T. Ward, and W.-J Zhu. BLEU: Method for Automatic Evaluation of Machine Translation. In Proc. of ACL, pages 311-318, 2002.
[45] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models Unsupervised Multitask Learners. OpenAl blog, 1(8):9, 2019.
[46] H. Rashkin, A. Celikyilmaz, Y. Choi, and J. Gao. PlotTMachines: OutlineCoonditonned Generation with Dynamic Plot State Tracking. arXiv Preprint, 2020.
12

---

[47] A. Sablayrolles. M. Douze, C. Schmid, and H. Jégou. Spreading vectors for similarity search.
In Proc. of ICLR, 2019.
[48] A. B. Sai, A. K. Mohankumar, and M. M. Khapra. A Survey of Evaluation Metrics Used for NLG Systems. arXiv Preprint, 2020.
[49] M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In Proc. of NeurIPS, 2018.
[50] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved Techniques for Training GANs, 2016.
[51] T. Sellam, D. Das, and A. P. Parikh. BLEURT: Learning Robust Metries for Text Generation.
In Proc. of ACL, pages 7881-7892, 2020.
[52] S. Semeniuta, A. Severyn, and S. Gelly. On Accurate Evaluation of GANs for Language Generation, 2018. arXiv Preprint.
[53] H. Shimanaka. T. Kajiwara, and M. Komachi. RUSE: Regressor Using Sentence Embeddingsfor
Automatic Machine Translation Evaluation. In Proc. of Conference on Machine Translation,
pages 751-758, 2018.
[54] A. Shimorina and A. Belz. The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP. arXiv Preprint, 2021.
[55] C. Tao, L. Mou, D. Zhao, and R. Yan. RUBER: An Unsupervised Method for Automatic Evaluation Open-Domain Dialog Systems. In Proc. of AAAI, 2018.
[56] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and
I. Polosukhin. Attention is All you Need. In Proc. of NeurlPS, pages 5998-6008, 2017.
[57] C. Villani. Topics in Optimal Transportation, volume 58. American Mathematical Soc., 2021.
[58] S. Welleck, I. Kulikov, J. Kim, R. Y. Pang, and K. Cho. Consistency of Recurrent Language Model With Respect to Incomplete Decoding. In Proc. of EMNLP, pages
2020.
[59] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural Text Generation With Unlikelihood Training. Proc. of ICLR, 2020.
[60] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jemite, J. Plu, C. Xu, T. L. Scao,
S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-Art Natural Language Processing. In Proc. of EMNLP, pages 38-45, 10 2020.
[61] R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, Roesner, and Y. Choi. Defending Against Neural Fake News. In Proc. of NeurlPS, 2019.
[62] H. Zhang, D. Duckworth, D. Ippolito, and A. Neelakantan. Trading off diversity and quality in natural language generation. Proc. of HumEval, pages 25-33, 2021.
[63] T. Zhang, Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi BERTScore: Evaluating text generation with BERT. In Proc. of ICLR, 2020.
[64] W. Zhao, M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, and S. Eger. MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In Proc. of EMNLP, 2019.
[65] Y. Zhu, S. Lu, L. Zheng, Guo, W. Zhang, J. Wang, and Yu. Texygen: A Benchmarking Platform for Text Generation Models, 2018.
13


Reviews:
[{"review_id": "kqlgS9OjSci", "reviewer": "Reviewer_wJx2", "summary": "This paper proposes an automated metric named Mauve for open-ended text generation to measure the gap between model-generated text and human text. This metric quantifies the divergence between the generated text distribution and human text distribution. It computes this by plotting the area under the divergence curve using KL-divergence. Given samples of generated text and human text, the authors propose to embed them using a pre-trained language model (GPT-2), assign the embeddings into clusters to finally compute the categorical distribution probabilities, which are then used to compute points in the divergence plot, the area under the curve produces the scalar value of the metric. The authors show that Mauve fares well in terms of capturing known properties of generated text like text length, model size, and decoding strategy and correlates well with human judgment. They also show that Mauve is robust to its various design choices like quantization, embeddings, etc.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points:\n1. Generalizes common divergences: forward/reverse KL, JSD.\n2. The soft weighted distribution in denominator makes sure that the points are never infinite like forward or reverse KL.\n3. Trade-off between Type 1 and Type 2 errors is very crucial in evaluating open-ended text generation where there is no correct reference text as in machine translation or summarization.\n4. Thorough comparison in related work.\n5. The idea is theoretically sound and motivationally strong. The paper is very well-written and clear in conveying the messages.\n\n6. Comprehensive analysis has been done on the following dimensions: \n* Tasks (different settings : web text, news and stories, each domain consists of sequence dataset split into context, continuation pairs).\n* Language mode: GPT-2 large and others like RoBERTa large.\n* Decoding strategies: ancestral sampling, nucleus sampling, and greedy.\n* Adversarial sampling: Generate low-quality text matching human text perplexity.\n\n7. Properties of Generated Text:\n* Mauve shows a negative correlation with generation length compared to other metrics compared - as expected.\n* Mauve identifies quality differences between decoding algorithms: greedy < ancestral < nucleus. Mauve is also robust to the adversarial decoding strategy.\n* Mauve shows a positive correlation with model size, agreeing with expectations, and with human quality assessments. For ancestral sampling, Mauve shows that a small model is better than a medium model, agreeing with human evaluators.\n\n8. Compared to other metrics which measure a single statistic or a single point on the divergence curve, Mauve captures a holistic picture considering a lot of points in the divergence curve.\n\n9. Design choices of Mauve does not affect the analyses presented - shows robustness:\n* Embeddings: RoBERTa vs GPT-2 : similar trends.\n* Quantization: k-means, DRMM, Lattice Quantization: rankings are similar.\n* Scaling parameter c: Doesn't affect the relative order of the divergence curve, though c=5 turns out to yield interpretable values.\n\n10. Additional points which favour the proposed metric:\n* Mauve correlates well with human judgments compared to other automatic metrics. So, costly human evaluation can be replaced with Mauve.\n* Mauve correlates well with learned discriminators. Also, Mauve doesn't need training, so can be used instead.\n\n11. The impacts statement is well articulated.\n\n\n## Concerns:\n1. Not an absolute metric: dependent upon the language model used.\n2. Also, there is a mismatch between the domain where GPT-2 is trained on and between the domains/tasks considered here. So, the question remains whether GPT-2 alone is appropriate.\n3. Choice of metrics to compare with varies between property demonstration: like Gen ppl, sparsemax, Frechet used for generation length analysis and whereas a bunch of others for comparing decoding algorithm. It's not clear why so.\n4. In decoding strategies, including beam search based strategy for comparison would have made the already detailed analyses complete.\n5. Typo line 208: *rates rates*.\n\n## Originality: 7/10 \nGiving moderate score as I am not fully aware of recent works in this area.\n\n## Quality: 8/10 \nSound motivation and theory, thorough analysis barring a few questions as stated above.\n\n## Clarity: 9/10. \nPaper is very well-written.\n\n## Significance: 8/10 \nAn important problem to solve, text generation evaluation is hard, especially open-ended text generation. This work with its plethora of positive analyses and without the need for any costly evaluation or training makes it a significant step in the positive direction for text generation evaluation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have clearly stated the implications in the broader impact statement.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "g9hg8CpTEF-", "reviewer": "Reviewer_9kXE", "summary": "This paper proposes comparison metrics for open-ended text generation. The method, named Mauve, compares the model’s distribution against the distribution of human-written text. \nThere are two types of errors identified and established to motivate the metric. The main idea of the metric is to measure the mutual distance from P to Q and Q to P. Due to the computational tractability issue, an approximated version based on quantized discrete version of the measure is proposed. Experiments show that Mauve agrees well with human judgment compared to competitor methods. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an interesting problem: automatically measuring machine generated text in the era of pre-trained LMs. The idea itself is not groundbreaking, but it makes sense to me and I would say it’s well executed. Distributional metric is an active research area due to the emerge of LMs, and this paper is a solid step towards this direction. \n\nExperiments are conducted on various models and various model sizes. There is also a good list of comparison models. \n\nThe code release and library contribution is also decent. It’s crucial for this kind of tool paper. \nThe presentation of the paper is good overall. I can understand most of the paper. All of the figures are color blind friendly and well captioned. \n\nQuestions:\nHow is the distance actually computed? There are two clusters with potentially different k selected. if c(x) returns the nearest cluster, how are clusters represented? I got a little confused from 102 - 113.  Some setup is needed. Maybe an illustration diagram is even better. \n\n“Mauve summarizes all of the divergence curve C(P,Q)” How does that work? Integration of all lambda from 0 to 1? Is there only one curve or infinite many curves? If you tune the lambda, C(P,Q) should form a single path, right?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Curve\nThe main selling point of this paper is the divergence curve. I am not 100% sure if I understand the concept, but it seems that what matters is basically computing KL(Q|P) and KL(P|Q), or KL(Q|R) and KL(P|R). If the idea is to compute the distance of two distributions, why do we care about the curve then? What's the role of lambda here in the curve?\n\nComparison\nMany of the comparison experiments are done to show how does different model size matter, if you skim the legends of most figures. I think there lacks in-depth analysis of comparison with existing metrics, like earth-mover distance, HUSE, or some baselines. This weakens the paper since we don’t know the advantage of this method against existing moving distance metrics and why we should use this instead of existing ones. \nAlso, I think more examples are needed to illustrate the advantages and disadvantages of the methods. I also want to know how the method generalizes to close-domain tasks like summarization.\n\nComplicatedness\nThe metric is more complicated due to all the KL, lambda, quantization. I have tried to measure the distance between two LM output distributions, and I found some numerical instability (the distance is not bounded, the distance does not reflect human insight, etc.). I ended up with just measuring the prob mass needed to be moved from two distributions, which is a simplified version of earth mover. I think this method is one more level of complication than earth mover but the core is the same. \n\n“Source of error in text generation” is not well supported and grounded\nAt the beginning of this paper, the authors motivate the method by talking about Type Errors (Figure 1) and Section 2. I think the visualization is very nice for people to understand the intuition and get excited about the paper. However, I am not sure if this is the case, and I also don’t see much analysis and support for this claim. \nFirst, the two dimensional figure is very \"abstract\" and I want some real statistics if possible. I personally have done some work using some distributional metrics, and I don’t feel the story is simple enough to be categorized as Type I and Type II. The shape of the distribution could be very complicated and different w.r.t. different types of generation. Second, the paper never shows any evidence for Type I and Type II errors. It seems that the paper could totally go without Type I and II errors. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "VM12b9r1cuf", "reviewer": "Reviewer_RwVW", "summary": "This paper introduces a method to quantify closeness in terms of  information divergence between a distribution of text sampled (generated) from a language model to human written text. These types of measures are useful in evaluating language models in open text generation tasks, but also could be extended for automatic detection of machine generated text.   \n\nThe metric that the paper introduces is \"Mauve\", theoretically it is the AOC  of a plot composed of the fwd KL divergence $KL(P||R_\\lambda)$ and  $KL(Q||R_\\lambda)$ where $P$ is the model distribution , $Q$ is the human text distribution and $R_\\lambda$ is the a mixture distribution of both (using different values of a mixing parameter $\\lambda$).\n\nUnlike different information divergence metrics  (e.g. vanilla KL or Jensen–Shannon) Mauve captures more general information about the divergence between the distributions. By analogy: in standard discrimination tasks ROC can capture the tradeoff between precision (Type 1 errors) and recall (type 2 errors), Similarly Mauve could capture Type 1 and Type 2 divergences than the human text distribution. \nThis is interesting as for example, due to its capacity of capturing type 2 errors, it can inherently asses the diversity of the generated text, covering information that can be captured by the self-bleu metric.  \n\nNow Approximating such KL values in the Mauve metrics is challenging in practice. Even when relying on Montecarlo estimation of to sample from $x \\sim P(x)$  , Most of the sampled x from the model will not exist in the large dataset of human written text Q(x) therefore leading to inf values of KL divergence. \n\nInstead authors propose to convert the P,Q from distribution over a high dimensional space (seq of text) to a multinomial distribution. This is done by converting discrete text into continuous embedding then binning them into 1 of k clusters. While this transformation doesn't guarantee unbiasedness in the KL estimations it allows tractability of calculations. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The overall paper is well written and clear. \n- The use of area under the curve is original for evaluation of text generation, the connection with type 1 and type 2 errors is quite interesting\n- While I had lots of skeptism on the robustness of the binning method and estimation of KL in a tractable method, authors provide a extensive comparison of different binning and embedding methods showing its robustness.  \n\nPossible Enhancements & Questions:\nQ1 - I have doubts on the comparability of Mauve scores across several runs? Given the same setup, different sampled batches from the Model P(x) could yield to completely different clusters, this could yield different high variance in Mauve scores, do you have comments on std of Mauve score calculation across several runs? \nQ2- (similar to Q1) but given I have Model 1 and Model 2 in comparison, both could yield different clusters making Mauve scores incomparable. I wonder if the clustering algorithm (e.g. Kmeans) should be fitted only on the human text, to provide more comparability between Mauve scores of different models?\n\n\nReferences:\n\nBelow some works that are not directly aimed at building evaluation metrics but in each of their evaluation respectively they use some relevant evaluation techniques that has some similarities with Mauve. \n\n- Assessment of Quality vs Diversity of the generated text over a whole curve with sweeping param instead of single pointwise comparison.  https://arxiv.org/abs/1811.02549\n- In the valuation an MC estimation to calculate KL divergence between text distributions https://arxiv.org/abs/2012.11635\n- Using hierarchical Bayesian models to compare the distribution of statistics extracted from generated text vs human gold references for MT\nhttps://aclanthology.org/2020.coling-main.398.pdf", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Similar to any evaluation metric automatically detecting human generation text, there's the risk of dual use. One could use such metrics as an optimization objective yielding generated text that is harder to differentiate from human written ones with many negative applications. Authors elaborate such risks clearly in the boarder impact statement in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "BfUV-EPpas", "reviewer": "Reviewer_Pk1q", "summary": "This paper presents a framework for evaluating open-ended text generation models: that is, evaluating how closely they match the distribution of human text (P). The proposed method (Mauve) evaluates for both Type I errors (false positives) and Type II errors (false negatives).  Mauve measures the divergence between P and Q by looking at the Pareto frontier of distributions R that are close to P and Q: that is, looking at the curve KL(P|R), KL(Q|R), which is optimized by R being an interpolation of P and Q.  This curve cannot be computed exactly since the true text distribution P is unknown. Mauve therefore takes samples, embeds them, clusters them, and approximates P and Q as multinomial distributions over the clusters, allowing us to interpolate their distributions and take KL divergences.\n\nResults show that this metric (1) has some desirable properties like rating long text as worse (since it often is in these settings) and reproducing human judgments about different sampling schemes; (2) is robust and works with different hyperparameters; (3) correlates well with human judgments.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a nice paper. The metric presented is theoretically well-motivated and shows some good results at evaluation of open-ended generation. This specific evaluation setting is not one that's seen a ton of specialized work (most generation evaluation focuses on comparing to one or more references), but having a metric for open-ended generation will definitely improve rigor.  I can also see Mauve enabling new work by researchers who are perhaps scared off by the focus on rigorous human evaluation in this setting.\n\nThe results are quite strong, although I'll qualify that by saying that most of the baseline metrics are not super well-suited to this task (with the exception of perplexity, but even that isn't seriously used as a holistic evaluation method by most work I've seen). Regardless, the absolute correlations with human judgments are high.\n\nOriginality\n\nConceptually, this work has some strong similarities to HUSE. HUSE formulates a discrimination problem and approaches the estimation of the true distribution P quite differently (and I don't love what HUSE does), but I felt that HUSE was not really highlighted strongly enough in the present work. Regardless, I agree with the paper's claim that a fully automatic metric is superior to a human-in-the-loop one. I am also prepared to accept the claim that this is a better-motivated metric than others that have attempted to balance precision and recall for generation.\n\nQuality\n\nThis paper is very high quality. The level of rigor is high, the experiments are carefully chosen and test important aspects of the metric's performance.  The only thing I would've liked to see more of would be some qualitative evaluation of the clustering procedure. I realize the paper had to prioritize other experiments in the interest of space, but it would've been helpful to understand (a) what the samples look like; (b) something about the contents of the clusters.\n\nClarity\n\nWhile I don't really hold this too seriously against the presentation of the paper, since everything is correctly defined and clear to me\nnow, I found myself struggling to understand the clustering and approximation procedure for some reason. I was hung up on the approximation of\n\n$$P(x) \\approx \\tilde{P}(c(x))$$\n\nFor some reason I interpreted this as a continuous distribution the first time around, although I see now that it's a distribution over clusters. But I find the idea that $\\tilde{P}$ \"approximates\" P to be sort of weird.\n\nI also found C_P/C_Q to be sort of hard to navigate, but this is probably just my fault. Equation 2 could've used a plain language gloss though. What would really help is pseudocode for the method (basically expanding compute_mauve in Appendix B). The procedure here seems quite natural: sample from both P and Q, cluster, count to form histograms, and then compute KL between the two discrete distributions.  This was harder to understand than it should've been.\n\nSignificance: This paper has the potential to be quite significant. I can see Mauve becoming a standard evaluation metric in these settings.\n\nMINOR COMMENTS\n\nWhat are the sampled points? Should these provide broad coverage or basically cover some \"canonical\" utterances in a domain of interest?  With these models having such flat distributions over the space, I wonder if there's any benefit to having some kind of proposal distribution here.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This has been addressed adequately.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Mauve, a novel metric for evaluating open-ended text generation. The paper is exceptionally well written, the idea makes a lot of sense and the experimental results are comprehensive, thorough and convincing. The reviewers unanimously agree that this is a clear accept.
Decision: Accept (Oral)