Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes an automated metric named Mauve for open-ended text generation to measure the gap between model-generated text and human text. This metric quantifies the divergence between the generated text distribution and human text distribution. It computes this by plotting the area under the divergence curve using KL-divergence. Given samples of generated text and human text, the authors propose to embed them using a pre-trained language model (GPT-2), assign the embeddings into clusters to finally compute the categorical distribution probabilities, which are then used to compute points in the divergence plot, the area under the curve produces the scalar value of the metric. The authors show that Mauve fares well in terms of capturing known properties of generated text like text length, model size, and decoding strategy and correlates well with human judgment. They also show that Mauve is robust to its various design choices like quantization, embeddings, etc.
main_review: ## Strong points:
1. Generalizes common divergences: forward/reverse KL, JSD.
2. The soft weighted distribution in denominator makes sure that the points are never infinite like forward or reverse KL.
3. Trade-off between Type 1 and Type 2 errors is very crucial in evaluating open-ended text generation where there is no correct reference text as in machine translation or summarization.
4. Thorough comparison in related work.
5. The idea is theoretically sound and motivationally strong. The paper is very well-written and clear in conveying the messages.

6. Comprehensive analysis has been done on the following dimensions: 
* Tasks (different settings : web text, news and stories, each domain consists of sequence dataset split into context, continuation pairs).
* Language mode: GPT-2 large and others like RoBERTa large.
* Decoding strategies: ancestral sampling, nucleus sampling, and greedy.
* Adversarial sampling: Generate low-quality text matching human text perplexity.

7. Properties of Generated Text:
* Mauve shows a negative correlation with generation length compared to other metrics compared - as expected.
* Mauve identifies quality differences between decoding algorithms: greedy < ancestral < nucleus. Mauve is also robust to the adversarial decoding strategy.
* Mauve shows a positive correlation with model size, agreeing with expectations, and with human quality assessments. For ancestral sampling, Mauve shows that a small model is better than a medium model, agreeing with human evaluators.

8. Compared to other metrics which measure a single statistic or a single point on the divergence curve, Mauve captures a holistic picture considering a lot of points in the divergence curve.

9. Design choices of Mauve does not affect the analyses presented - shows robustness:
* Embeddings: RoBERTa vs GPT-2 : similar trends.
* Quantization: k-means, DRMM, Lattice Quantization: rankings are similar.
* Scaling parameter c: Doesn't affect the relative order of the divergence curve, though c=5 turns out to yield interpretable values.

10. Additional points which favour the proposed metric:
* Mauve correlates well with human judgments compared to other automatic metrics. So, costly human evaluation can be replaced with Mauve.
* Mauve correlates well with learned discriminators. Also, Mauve doesn't need training, so can be used instead.

11. The impacts statement is well articulated.


## Concerns:
1. Not an absolute metric: dependent upon the language model used.
2. Also, there is a mismatch between the domain where GPT-2 is trained on and between the domains/tasks considered here. So, the question remains whether GPT-2 alone is appropriate.
3. Choice of metrics to compare with varies between property demonstration: like Gen ppl, sparsemax, Frechet used for generation length analysis and whereas a bunch of others for comparing decoding algorithm. It's not clear why so.
4. In decoding strategies, including beam search based strategy for comparison would have made the already detailed analyses complete.
5. Typo line 208: *rates rates*.

## Originality: 7/10 
Giving moderate score as I am not fully aware of recent works in this area.

## Quality: 8/10 
Sound motivation and theory, thorough analysis barring a few questions as stated above.

## Clarity: 9/10. 
Paper is very well-written.

## Significance: 8/10 
An important problem to solve, text generation evaluation is hard, especially open-ended text generation. This work with its plethora of positive analyses and without the need for any costly evaluation or training makes it a significant step in the positive direction for text generation evaluation.
limitations_and_societal_impact: The authors have clearly stated the implications in the broader impact statement.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a framework for evaluating open-ended text generation models: that is, evaluating how closely they match the distribution of human text (P). The proposed method (Mauve) evaluates for both Type I errors (false positives) and Type II errors (false negatives).  Mauve measures the divergence between P and Q by looking at the Pareto frontier of distributions R that are close to P and Q: that is, looking at the curve KL(P|R), KL(Q|R), which is optimized by R being an interpolation of P and Q.  This curve cannot be computed exactly since the true text distribution P is unknown. Mauve therefore takes samples, embeds them, clusters them, and approximates P and Q as multinomial distributions over the clusters, allowing us to interpolate their distributions and take KL divergences.

Results show that this metric (1) has some desirable properties like rating long text as worse (since it often is in these settings) and reproducing human judgments about different sampling schemes; (2) is robust and works with different hyperparameters; (3) correlates well with human judgments.
main_review: This is a nice paper. The metric presented is theoretically well-motivated and shows some good results at evaluation of open-ended generation. This specific evaluation setting is not one that's seen a ton of specialized work (most generation evaluation focuses on comparing to one or more references), but having a metric for open-ended generation will definitely improve rigor.  I can also see Mauve enabling new work by researchers who are perhaps scared off by the focus on rigorous human evaluation in this setting.

The results are quite strong, although I'll qualify that by saying that most of the baseline metrics are not super well-suited to this task (with the exception of perplexity, but even that isn't seriously used as a holistic evaluation method by most work I've seen). Regardless, the absolute correlations with human judgments are high.

Originality

Conceptually, this work has some strong similarities to HUSE. HUSE formulates a discrimination problem and approaches the estimation of the true distribution P quite differently (and I don't love what HUSE does), but I felt that HUSE was not really highlighted strongly enough in the present work. Regardless, I agree with the paper's claim that a fully automatic metric is superior to a human-in-the-loop one. I am also prepared to accept the claim that this is a better-motivated metric than others that have attempted to balance precision and recall for generation.

Quality

This paper is very high quality. The level of rigor is high, the experiments are carefully chosen and test important aspects of the metric's performance.  The only thing I would've liked to see more of would be some qualitative evaluation of the clustering procedure. I realize the paper had to prioritize other experiments in the interest of space, but it would've been helpful to understand (a) what the samples look like; (b) something about the contents of the clusters.

Clarity

While I don't really hold this too seriously against the presentation of the paper, since everything is correctly defined and clear to me
now, I found myself struggling to understand the clustering and approximation procedure for some reason. I was hung up on the approximation of

$$P(x) \approx \tilde{P}(c(x))$$

For some reason I interpreted this as a continuous distribution the first time around, although I see now that it's a distribution over clusters. But I find the idea that $\tilde{P}$ "approximates" P to be sort of weird.

I also found C_P/C_Q to be sort of hard to navigate, but this is probably just my fault. Equation 2 could've used a plain language gloss though. What would really help is pseudocode for the method (basically expanding compute_mauve in Appendix B). The procedure here seems quite natural: sample from both P and Q, cluster, count to form histograms, and then compute KL between the two discrete distributions.  This was harder to understand than it should've been.

Significance: This paper has the potential to be quite significant. I can see Mauve becoming a standard evaluation metric in these settings.

MINOR COMMENTS

What are the sampled points? Should these provide broad coverage or basically cover some "canonical" utterances in a domain of interest?  With these models having such flat distributions over the space, I wonder if there's any benefit to having some kind of proposal distribution here.
limitations_and_societal_impact: This has been addressed adequately.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes comparison metrics for open-ended text generation. The method, named Mauve, compares the model’s distribution against the distribution of human-written text. 
There are two types of errors identified and established to motivate the metric. The main idea of the metric is to measure the mutual distance from P to Q and Q to P. Due to the computational tractability issue, an approximated version based on quantized discrete version of the measure is proposed. Experiments show that Mauve agrees well with human judgment compared to competitor methods. 

main_review: The paper tackles an interesting problem: automatically measuring machine generated text in the era of pre-trained LMs. The idea itself is not groundbreaking, but it makes sense to me and I would say it’s well executed. Distributional metric is an active research area due to the emerge of LMs, and this paper is a solid step towards this direction. 

Experiments are conducted on various models and various model sizes. There is also a good list of comparison models. 

The code release and library contribution is also decent. It’s crucial for this kind of tool paper. 
The presentation of the paper is good overall. I can understand most of the paper. All of the figures are color blind friendly and well captioned. 

Questions:
How is the distance actually computed? There are two clusters with potentially different k selected. if c(x) returns the nearest cluster, how are clusters represented? I got a little confused from 102 - 113.  Some setup is needed. Maybe an illustration diagram is even better. 

“Mauve summarizes all of the divergence curve C(P,Q)” How does that work? Integration of all lambda from 0 to 1? Is there only one curve or infinite many curves? If you tune the lambda, C(P,Q) should form a single path, right?

limitations_and_societal_impact: Curve
The main selling point of this paper is the divergence curve. I am not 100% sure if I understand the concept, but it seems that what matters is basically computing KL(Q|P) and KL(P|Q), or KL(Q|R) and KL(P|R). If the idea is to compute the distance of two distributions, why do we care about the curve then? What's the role of lambda here in the curve?

Comparison
Many of the comparison experiments are done to show how does different model size matter, if you skim the legends of most figures. I think there lacks in-depth analysis of comparison with existing metrics, like earth-mover distance, HUSE, or some baselines. This weakens the paper since we don’t know the advantage of this method against existing moving distance metrics and why we should use this instead of existing ones. 
Also, I think more examples are needed to illustrate the advantages and disadvantages of the methods. I also want to know how the method generalizes to close-domain tasks like summarization.

Complicatedness
The metric is more complicated due to all the KL, lambda, quantization. I have tried to measure the distance between two LM output distributions, and I found some numerical instability (the distance is not bounded, the distance does not reflect human insight, etc.). I ended up with just measuring the prob mass needed to be moved from two distributions, which is a simplified version of earth mover. I think this method is one more level of complication than earth mover but the core is the same. 

“Source of error in text generation” is not well supported and grounded
At the beginning of this paper, the authors motivate the method by talking about Type Errors (Figure 1) and Section 2. I think the visualization is very nice for people to understand the intuition and get excited about the paper. However, I am not sure if this is the case, and I also don’t see much analysis and support for this claim. 
First, the two dimensional figure is very "abstract" and I want some real statistics if possible. I personally have done some work using some distributional metrics, and I don’t feel the story is simple enough to be categorized as Type I and Type II. The shape of the distribution could be very complicated and different w.r.t. different types of generation. Second, the paper never shows any evidence for Type I and Type II errors. It seems that the paper could totally go without Type I and II errors.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a method to quantify closeness in terms of  information divergence between a distribution of text sampled (generated) from a language model to human written text. These types of measures are useful in evaluating language models in open text generation tasks, but also could be extended for automatic detection of machine generated text.   

The metric that the paper introduces is "Mauve", theoretically it is the AOC  of a plot composed of the fwd KL divergence $KL(P||R_\lambda)$ and  $KL(Q||R_\lambda)$ where $P$ is the model distribution , $Q$ is the human text distribution and $R_\lambda$ is the a mixture distribution of both (using different values of a mixing parameter $\lambda$).

Unlike different information divergence metrics  (e.g. vanilla KL or Jensen–Shannon) Mauve captures more general information about the divergence between the distributions. By analogy: in standard discrimination tasks ROC can capture the tradeoff between precision (Type 1 errors) and recall (type 2 errors), Similarly Mauve could capture Type 1 and Type 2 divergences than the human text distribution. 
This is interesting as for example, due to its capacity of capturing type 2 errors, it can inherently asses the diversity of the generated text, covering information that can be captured by the self-bleu metric.  

Now Approximating such KL values in the Mauve metrics is challenging in practice. Even when relying on Montecarlo estimation of to sample from $x \sim P(x)$  , Most of the sampled x from the model will not exist in the large dataset of human written text Q(x) therefore leading to inf values of KL divergence. 

Instead authors propose to convert the P,Q from distribution over a high dimensional space (seq of text) to a multinomial distribution. This is done by converting discrete text into continuous embedding then binning them into 1 of k clusters. While this transformation doesn't guarantee unbiasedness in the KL estimations it allows tractability of calculations. 

main_review: Pros:
- The overall paper is well written and clear. 
- The use of area under the curve is original for evaluation of text generation, the connection with type 1 and type 2 errors is quite interesting
- While I had lots of skeptism on the robustness of the binning method and estimation of KL in a tractable method, authors provide a extensive comparison of different binning and embedding methods showing its robustness.  

Possible Enhancements & Questions:
Q1 - I have doubts on the comparability of Mauve scores across several runs? Given the same setup, different sampled batches from the Model P(x) could yield to completely different clusters, this could yield different high variance in Mauve scores, do you have comments on std of Mauve score calculation across several runs? 
Q2- (similar to Q1) but given I have Model 1 and Model 2 in comparison, both could yield different clusters making Mauve scores incomparable. I wonder if the clustering algorithm (e.g. Kmeans) should be fitted only on the human text, to provide more comparability between Mauve scores of different models?


References:

Below some works that are not directly aimed at building evaluation metrics but in each of their evaluation respectively they use some relevant evaluation techniques that has some similarities with Mauve. 

- Assessment of Quality vs Diversity of the generated text over a whole curve with sweeping param instead of single pointwise comparison.  https://arxiv.org/abs/1811.02549
- In the valuation an MC estimation to calculate KL divergence between text distributions https://arxiv.org/abs/2012.11635
- Using hierarchical Bayesian models to compare the distribution of statistics extracted from generated text vs human gold references for MT
https://aclanthology.org/2020.coling-main.398.pdf
limitations_and_societal_impact: Similar to any evaluation metric automatically detecting human generation text, there's the risk of dual use. One could use such metrics as an optimization objective yielding generated text that is harder to differentiate from human written ones with many negative applications. Authors elaborate such risks clearly in the boarder impact statement in the paper.
review_title: null

Meta-review: This paper proposes Mauve, a novel metric for evaluating open-ended text generation. The paper is exceptionally well written, the idea makes a lot of sense and the experimental results are comprehensive, thorough and convincing. The reviewers unanimously agree that this is a clear accept.
Decision: Accept (Oral)