BARTScore: Evaluating Generated Text as Text Generation
Authors: Weizhe Yuan
Graham Neubig
Pengfei Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Pre-trained models
Text Generation
Sequence-to-Sequence

Abstract: A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.

BARTSCORE:
Evaluating Generated Text as Text Generation
Weizhe Yuan Carnegie Mellon University weizhey8cs cmu. edu
Graham Neubig Carnegie Mellon University gneubigles, cmu edu
Pengfei Liu
Carnegie Mellon University pliu30cs cmu. edu
Abstract
A wide variety of NLP applications, such machine translation. summarization, and dialog, involve text generation. One major challenge for these applications is how evaluate whether such generated texts actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text text generation problem, modeled using pre-trained sequence -o-enceeee models. The general idea is that models trained to convert the generated text to/from reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART [32], an ricoder-decoder based pre-trained model, and propose a metric BARTSCORE with number of variants that can be flexibly applied in an unsupervised fashion evaluation of text from different perspectives (c.g. informativeness, fluency, factuality). BARTSCORE is conceptually simple and empirically effective. It can outperform existing top-scoring metries in 16 of 22 test settings, cov- ering evaluation of 16 datasets (e.g.. machine translation, text summarization) and different perspectives (e.g., informativenesss factuality). Code to calculate BARTScore is available at https:: / gi thub........ com BeeeuuabbBBRRRR and we have released an interactive leaderboard for meta-evaluation at http: / //explainabon rd... 1 pedia a /i//1 /2//ee /eaddeeeeeeee (board/ta ttstt/ssskmallll on the EXPLAINABOARD platform [38], which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
Introduction
One defining feature of recent NLP models is the use of neural representations trained on raw text,
using unsupervised objectives such language modeling [6,53], denoising autoencoding [9.32.54].
By learning to predict the words or sentences in natural text, these models simultaneously learn to
extract features that not only benefit mainstream NI.P tasks such as information extraction [23,37] 37],
question answering [1, 162 text summarization [40,77] but also have proven effective in development of automatic metrics for evaluation of text generation itself [62, 65). For example, BERTScore [75]
and MoverScore [76] take features extracted by BERT [9] and apply unsupervised matching functions to compare system outputs against references. Other works build supervised frameworks that use extracted features to learn to rank [56] or regress [62] to human evaluation scores.
However, in the context of generation evaluation, one may note that there is decided disconnect between how models are pre-trained using text generation objectives and how they are used as down-stream feature extractors. This leads to potential under-utiizztion of the pre-trained model parameters. For example, the output prediction layer is not used at all in this case. This disconnect particularly striking because of the close connection between the pre-training objectives and the generation tasks we want to evaluate.
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we instead argue for formulation of evaluation of generated text as text generation problem, directly evaluating text through the lens of its probability of being generated from generat- ing other textual inputs and outputs. This is a better match with the underlying pre-training tasks and allows us to more fully take advantage of the parameters learned during the pre-training phase. We solve the modeling problem with a pre-trained eqquence-to-sqqeene (seq2seq) model, specifically BART [32], and devise a metric named B BARTSCORE, which has the following characteristics: (1) BARTSCORE is parameter- and data-efficient. Architecturally there are no extra parameters beyond those used pre-training itself, and it an unsupervised metric that doesn' 't require human judgments to train. (2) BARTSCORE can better support evaluation of generated text from different perspectives (e.g., informativeness, coherence, factuality, $4) by adjusting the inputs and outputs of the conditional text generation problem, mostly examines correlation of the devised metrics with output quality from a limited number of perspectives. (3) BARTSCORE can be further enhanced by (i) providing textual prompts that bring the evaluation task closer to the pre-training task, or (ii) updating the underlying model by fine-tuning BART based downstream generation tasks (e.g., text summarization).
demonstrate in $3.2. This is in contrast to most previous work, which
Experimentally, we evaluate different variants of BARTSCORE from perspectives on 16 datasets. BARTSCORE achieves the best performance in 16 of 22 test settings against existing top-scoring metrics. Empirical results also show the effectiveness of the prompting strategy supported by BARTSCORE. For example, simply adding the phrase "such as" to the translated text when using BARTSCORE can lead to a 3% point absolute improvement in correlation on "German -agunnsh"ih"" machine translation (MT) evaluation. Additional analysis shows that BARTSCORE is more robust when dealing with high-quality texts generated by rop-performing systems.
Preliminaries
2.1 Problem Formulation
As stated above, our goal is to assess the quality of generated text [3,46]. In this work, we focus on conditional text generation (e.g., machine translation), where the goal is to generate a hypothesis (h h,.... hhm) based on a given source text (s 81," ... 8x). Commonly, one or multiple human-created references (r "","" (1) are provided to aid this evaluation.
2.2 Gold-standard Human Evaluation
In general, the gold-standard method for evaluating such texts is still human evaluation, where human annotators assess the generated texts' quality. This evaluation can be done from perspectives, and we list a few common varieties below (all investigated in $4):
1. Informativeness (INFO): How well the generated hypothesis captures the key ideas of the source text [18].
2. Relevance (REL): How consistent the generated hypothesis is with respect to the source
text [19].
3. Fluency (FLU): Whether the text has no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read [13].
4. Coherence (COH): Whether the text builds from sentence to sentence to coherent body of
information about a topic [7].
5. Factuality (FAC): Whether the generated hypothesis contains only statements entailed by
the source text [30].
6. Semantic Coverage (Cov): How many semantic content units from reference texts
covered by the generated hypothesis [49].
7. Adequacy (ADE): Whether the output conveys the same meaning the input sentence, and none of the message is lost, added, or distorted [29].
Most existing evaluation metrics were designed to cover a small subset of these perspectives. For example, BLEU [50] aims to capture the adequacy and fluency of translations, while ROUGE [36] was designed to match the semantic coverage metric. Some metrics, particularly trainable ones, can perform evaluation from different perspectives but generally require maximizing correlation with each type of judgment separately [8].

---

5
$2
14
14
h2
1/2
32
(a) Matching
(b) Regression
(c)
Ranking
(d) Generation
Figure I: Evaluation metrics different tasks, where 8, and ry represent source, hypothesis and reference words respectively.
As we describe more in $4, BARTSCORE can evaluate text from the great majority of these perspec- tives, significantly expanding its applicability compared these metrics.
2.3 Evaluation as Different Tasks
There is a recent trend that leverages neural models for automated evaluation different ways, shown in Fig. 1. We first elaborate their characteristics by highlighting differences in task formulation and evaluation perspectives.
T1: Unsupervised Matching. Unsupervised matching metrics aim to measure the semantic equiv- alence between the reference and hypothesis by using a token-level matching functions dis- tributed representation space, such BERTScore [75], MoverScore [76] discrete string space like ROUGE [35]. BLEU [50], CHRF [52]. Although similar matching functions can be used to assess the quality beyond semantic equivalence (e.g, factuality, relationship between source text and hypothesis), to our knowledge prior research has not attested to the capability of unsupervised matching methods in this regard; explore this further in our experiments (Tab. 5).
T2: Supervised Regression. Regression- models introduce a parameterized regression layer, which would be learned in supervised fashion to accurately predict human judgments. Examples include recent metries BLEURT [62]. COMET [56] and traditional metrics like S [51], VRM [21].
T3: Supervised Ranking. Evaluation can also be conceived as ranking problem, where the main idea to learn scoring function that assigns a higher score better hypotheses than to worse ones. Examples include COMET [56] and BEER [64], where COMET focuses the machine translation task and relies on human judgments tune parameters in ranking or regression layers, and BEER combines many simple features in a tunable linear model of MT evaluation metrics.
T4: Text Generation. In this work, we formulate evaluating generated text text generation task from pre-trained language models. The basic idea is that a high-quality hypothesis will be easily generated based on source reference text vice-versa. This has not been covered as extensively in previous work, with one notable exception being PRISM [65]. Our work differs from PRISM in several ways: (i) PRISM formulates evaluation as a paraphrasing task, whose definition that two texts are with the same meaning limits its applicable scenarios, like factuality evaluation in text summarization that takes source documents and generated summaries as input whose semantic space are different. (ii) PRISM trained model from scratch on parallel data while BARTSCORE based on open-sourced pre-trained seq2seq models. (iii) BARTSCORE supports prompt- learning [59, 63] which hasn't 't been examined in PRISM.
3 BARTScore
3.1 Sequence-to-Seqqeenc Pre-trained Models
Although pre-trained models differ along different axes, one of the main axes of variation is the training objective, with two main variants: language modeling objectives (e.g.. masked language modeling [9]) and seq2seq objectives [54]. In particular, seq2seq pre-trained models are particularly well-suited to conditioned generation tasks since they consist of both an encoder and a decoder, and predictions made auto-regressively [32]. In this work, operationalize our idea by using
3

---

BART [32] as our backbone due to its superior performance in text generation [12,42,71], We also report preliminary experiments comparing BART with T5 [54] and PEGASUS [74] in the Appendix.
Given seq2seq model parameterized by 0, a source sequence containing n tokens x = {II,.... , ] and a target sequence containing tokens y (Vi,'.. ,Um)- We can factorize the generation probability of y conditioned on x as follows:
p 0)
II p(y:lvc..x.0)
By exploring these probabilities, we design metrics that can gauge the quality of the generated text.
3.2 BARTScore
The most general form of our proposed BARTSCORE is shown in Eq. 2. where we use the weighted log probability of one text given another text x. The weights are used to put different emphasis on different tokens, which can be instantiated using different methods like Inverse Document Frequency (IDF) [25] etc. In our work, we weigh each token equally.
BARTSCORE
we log log
Due to its generation lask-based formulation and ability to utilize the entirety of BART's praited parameters, BARTSCORE can be flexibly used in different evaluation scenarios. We specifically present four methods for using EARTSCORE based on different generation directions, which are,
Faithfulness (s - h): from source document to hypothesis p(h 8, 0). This direction measures how likely it is that the hypothesis could be generated based on the source text. Potential application scenarios are factuality and relevance introduced in $2.2. This measure can also be used for estimating measures of the quality of only the target text, such as coherence and fluency ($2.2). Precision (r - h): from reference text to system-gent generated text p(h/r, 0). This direction assesses how likely the hypothesis could be constructed based on the gold reference and is suitable for the precision-focused scenario.
Recall (h- + r): from system-generatee generated text to reference text p(r (h, 0). This version quantifies how easily gold reference could be generated by the hypothesis and is suitable for pyramid-bant based evaluation (i.e., semantic coverage introduced in $2.2) in summarization task since pyramid score measures fine-grained Semantic Content Units (SCUs) [49] covered by gystem-generale texts. F score (r <-> h): Consider both directions and use the arithmetic average of Precision and Recall ones. This version can be broadly used to evaluate the semantic overlap (informativeness, adequacy detailed in $2.2) between reference texts and generated texts.
3.3 BARTScore Variants
We also investigate two extensions to BARTSCORE: (i) changing x and y through prompting, which can bring the evaluation task closer to the pre-trane pre-training task. (ii) changing / by considering different fine-tuning tasks, which can bring the pre-training domain closer to the evaluation task.
3.3.1 Prompt
Prompting practice of adding short phrases to the input output to encourage pre-trained models to perform specific tasks, which has been proven effective in several other NL.P scenarios [24,57,58, 60.63J...... The generative formulation of BARTSCORE makes relatively easy to incorporate these insights here as well; we name this variant BARTSCORE-PPRMPTT
Given prompt of tokens case we get x' = { ... Fn, 21, ... 21 and calculate the score based on this new source text using Eq.2. (ii) prepend it to the target text, getting y' {21. 24, V1. ... 99m). Then we can also use Eq.2 given the new target text.
{21, "., 21 ). can either (i) append it to the source text, in which
We have tried several other weighting schemes, including: (i) uniform weighting while ignoring stop words. (ii) IDF weighting. (iii) using the prior probability of each target token (calculated within the target sequence) as the weighting factor. However, none of those outperformed the uniform weighting scheme.

---

3.3.2 Fine-tuning Task
Different from BERT-based metrics, which typically use classification-basee tasks (e.g., natural language inference) [67] to fine-tune, BARTSCORE can be fine-tuned using generation ienerationbbased tasks, which will make the pre-training domain closer the evaluation task. In this paper, explore two downstream tasks. (1) Summarization. We use BART fine-tuned on CNNDM dataset [20], which is available off-the-shelf in Huggingface Transformers [70]. (2) Paraphrasing. We continue fine-tuning BART from (1) on P ra araaank2 dataset [22], which contains a large paraphrase collection. We used random subset of 30,000 data and fine-tuned for epoch with batch size of 20 and learning rate of 5e - We used two 2080'i GPUs, and the training time is less than one hour.
4 Experiment
This section aims to evaluate the reliability of different automated metrics, which is commonly achieved by quantifying how well different metrics correlate with human judgments using measures (e.g., Spearman Correlation [72]) defined below ($4.1.2).
4.1 Baselines and Datasets
4.1.1 Evaluation Metrics
We comprehensively examine metrics outlined in $2.3, which either require human judgments to train (i.e.. supervised metrics): COMET [56]. BLEURT [62], or are human judgment-free (i.e., unsupervised): BLEU [50] ROUGE-1 and ROUGE-2, ROUGE-L, CHRF [52], PRISM [65], MoverScore [76], BERTScore [75]. The detailed comparisons of those metrics can be found in Appendix. We use the official code for each metric.
4.1.2 Measures for Meta Evaluation
Pearson Correlation [15] measures the linear correlation between two sets of data. Spearman Correlation [72] assesses the monotonic relationships between two variables. Kendall's Tau [27] measures the ordinal association between two measured quantities. Accuracy, in our experiments, measures the percentage of correct ranking between factual texts and non- texts. We follow previous works in the choices of measures for different datasets to make a fair comparison.
4.1.3 Datasets
The datasets we use are summarized in Tab. 1. We
Table 1: A summary of tasks, datasets, and evaluation perspectives that have covered in our experiments. Explanation evaluation perspectives can be found in $2.2.
consider three different tasks: summarization (SUM),
machine translation (MT), and data-to-text (D2T).
Machine Translation We obtain the source lan- guage sentences, machine-translated texts and refer- ence texts from the WMT19 9 metrics shared task [44]. use the DARR corpus and consider 7 language pairs, which are de-en, fi fi-en, gu-en, kk-en, It-en, ru- en, zh-en.
Tasks
Datasets
Eval. Perspectives
REALSUM
Cov
SummEval
COH FAC FLU INFO
NeR1 8
COH FL.U REL. INFO
SUM
Text Summarization (1) REALS [4] is a meta- evaluation dataset for text summarization which mea- sures pyramid recall of each system-generated sum- mary. (2) SummEval [13] is collection of hu- man judgments of model-genen generated summaries on
Rank19
QAGS-C
FAC
QAGS-X
FI GU KK
MT
IT RU ZH
ADE FLU
the CNNDM dataset annotated by both expert judges and crowd-source workers. Each system generated summary is gauged through the lens of coherence, consistency, flency and relevance. (3) NeR1 8 The
BAGEL SFHOT SFRES
D2T
INFO
We rephrase the original "relevance" into "informativeness" and "consistency" into "factuality" based on the descriptions in their paper and definitions in $2.2.
5

---

NEWSROOM dataset [18] contains 60 articles with summaries generated by 7 different methods are annotated with human scores in terms of coherence, fluency, informativeness, relevance.
a collection of 373 triples of a source sentence with two summary sentences, one correct and one incorrect. (2) QAGS20 [66] collected 235 test outputs on CNNDM dataset from [16] and 239 test outputs on XSUM dataset [47] from BART fine-tuned on XSUM. Sentences in each summary are annotated with correctness scores w.r.t. factuality.
Factuality (1) Rank [14] is used to meta-evaluate factuality metrics.
Data to Text We consider the following datasets which target utterance generation for spoken dialogue systems. (1) BAGEL [45] provides information about restaurants. (2) SFHOT [69] provides information about hotels in San Francisco. (3) SFRES [69] provides information about restaurants in San Francisco. They contain 202, 398, and 581 samples respectively, each sample consists of one meaning representation, multiple references, and utterances generated by different systems.
4.2
Setup
4.2.1
Prompt Design
To perform prompting, we first need find proper prompts within search space. Instead of considering a large discrete search space [63]' or continuous search space [34], we use simple heuristics narrow our search space. In particular, we use manually devised seed prompts and gather paraphrases to construct our prompt set. The seed prompts and some examples of paraphrased prompts are shown in Tab. Details are listed in the Appendix.
Table 2: Seed prompts and examples of final prompts. "Number" denotes the size of our final prompt set that was acquired from the seed prompts.
Usage
Number
Seed
Example
h 70
in summary
in short, in a word, to sum up to rephrase it, that to say, i.e.
h r
34
in other words
4.2.2 Settings
Variants. We consider four variants of BARTSCORE, which are (1) BARTSCORE, which uses the vanilla BART; (2) BARTSCORE-CNN, which uses the BART fine-tuned on the summarization dataset
CNNDM; (3) BARTSCORE C-N--ANRAA where BART is first fine-tuned on CNNDM, then fine-tuned which is enhanced by adding prompts.
on FaraBank2. (4) BARTSCORE-PROMME
Selection of Prompts. For the summarization and data-to-test tasks, prompts designed for + h or all prompts designed for h <> r depending on the BARTScore usage chosen) in the prompt set by prefixing the decoder input and getting different generation scores (calculated by 2).2) for each hypothesis based on different prompts. We finally get the score for one hypothesis by taking the average of all its generation scores using different prompts [24]; details about prompt ensembling can be found in the Appendix). For the machine translation task, due to the more expensive computational cost brought by larger text sets, we first use WMT18 [43] development set to search for one best prompt and obtain the phrase "Such as", which is then used for the test language pairs.
all entries (either all
Selection of BARTScore Usage Although BARTSCORE can be used in different ways (shown in $3.2)), in different tasks, they can be chosen based on how targeted evaluation perspectives defined (described $2.2) as well as the types of tasks. Specifically, (i) For those datasets whose gold standard human evaluation are obtained based recall-based pyramid method, we adopt recall-based BARTSCORE (h r). (ii) For those datasets whose human judgments focus on linguistic quality (coherence, fluency) and factual correctness (factuality), or the source and hypothesis texts are in the same modality (i.c., language), we use faithfulness-basee
BARTSCORE (s + h). (iii) For data-to-text and machine translation tasks, to make fair comparison, we use BARTSCORE with the F-score version that other existing works [65] have adopted when evaluating generated texts.
4 WWe explored this first and found that discovered prompts led worse performance.
We use the website at tps: / /www. ippo com/ search for synonyms.

---

Table 3: Kendall's Tau correlation of different metrics WMT19 dataset. The highest correlation for each language pair achieved by unsupervised method is bold, and the highest correlation overall is underlined. Avg. denotes the average correlation achieved by metric across all language pairs.
de-en
fi-en
gu-en
kk-en
It-en
ru-en
zh-en
Avg.
SUPERVISED METHODS
BLEURT
0.174 0.219
0.374
0,313
0.372 0.378
0,388 0.405
0.220
0,436
0.325
COMET
0.369
0.316
0.226
0.462
0.339
UNSUPERVISED METHODS
BLEU
0.054
0.236
0.194
0.276
0.249
0.115
0.321
0.206
CHRF
0.123
0.292
0.240
0.323
0.304
0.177
0.371
0.261
PRISM
0.199
0.366
0.320
0.362
0.382
0.220
0.434
0.326
BERTScore
0.190
0.354
0.292
0.351
0,381
0.221
0.430
0.317
BARTSCORE
0.156
0.335
0,273
0.324
0.322
0.167
0.389
0.281
CNN
0.190
0,365
0,300
0,348
0,384
0.208
0,425
0.317
CNN + Para
0.205f
0.3701
0,316
0.378
0.3861
0.219
0.442f
0.331
CNN Para Prompt
0.238
0.374t
0.318
0.376
0.386
0.219
0.4477
0.337
Table 4: Spearman correlation of different metrics on three human judgement datasets. For prompt- based learning, we consider adding prompts the best-performing BARTSCORE (S) each dataset. The highest correlation overall for each aspect on each dataset is bold.
REALSumm
SummEval
NeR18
Cov
Con
FAC
FLU
INFO
COH
FL.U
INFO
REL.
Avg.
ROUGE-1
0.498
0.167
0.160
0.115
0.326
0.095
0.104
0.130
0.147
0.194
ROUGE-2
0.423
0.184
0.187
0.159
0,290
0.026
0.048
0.079
0.091
0.165
ROUGE-L
0,488
0.128
0.115
0.105
0.311
0.064
0.072
0.089
0.106
0.164
BERTScore
0.440
0.284
0.110
0.193
0.312
0.147
0.170
0.131
0.163
0.217
MoverScore
0.372
0.159
0.157
0.129
0.318
0.161
0.120
0.188
0.195
0.200
PRISM
0.411
0.249
0.345
0.254
0.212
0.573
0.532
0.561
0.553
0.410
BARTSCORE
0.441
0.322
0.311
0.248
0.264
0.679
0.670
0.646
0.604
0.465
CNN
0.475
0.448
0.3821
0.356t
0.356t
0.653t
0.640
0.616t
0.567
0.499
+ CNN Para
0.471
0.424
0.401
0.378
0.313
0.657f
0.652f
0.614f
0.562
0.497
+ Ql Prompt
0.488
0.407t
0.378
0.338
0.368
0.701
0.679;
0.686;
0.620
0.518
Significance Tests. To perform rigorous analysis, we adopt the bootstrapping method (p-value < 0.05) [28] for pair-wise significance tests. In all tables, we BARTSCORE it significantly (p < 0.05) outperforms other unsupervised metrics excluding BARTSCORE variants. We use | on BARTSCORE if it significantly outperforms all other unsupervised metries including BARTSCORF variants.
4.3 Experimental Results
4.3.1 Machine Translation
Tab. 3 illustrates Kendall's Tau correlation of diverse metrics on different language pairs. We can observe that: (1) BARTSCORE enhanced by fine-tuning tasks (CNN+Para) can significantly outperform all other unsupervised methods on five language pairs and achieve comparable results on the other two. (2) The performance of BARTSCORE can be further improved by simply adding prompt (i.e., such as) without any other overhead. Notably, on the language pair de-en, using the prompt results in a 0.033 improvement, which even significantly surpasses existing state-of-the-ee supervised metrics BLEURT and COMET. This suggests a promising future direction for metric design: searching for proper prompts to better leverage knowledge stored pre-trained language models instead of training human judgment data [31].
7

---

4.3.2 Text Summarization
Tab. 4 shows the meta-evaluation results of different metrics on the summarization task. We can observe that: (1) Simply vanilla BARTSCORE outperform BERTScore and MoverScore by a large margin on 8 settings except the INFO perspective on SummEval. Strikingly, it achieves improvements of 0.251 and 0.265 over BERTScore and MoverScore respectively. (2) The improve- ment on REALSum and SummEval datasets can be further improved when introducing fine-tuning tasks. However, fine-tuning does improve on the NeR18 18 dataset, likely because this dataset only contains systems with easily distinguishable quality, and vanilla BARTSCORE can already achieve high level of correlation (> 0.6 on average). (3) Our prompt combination strategy can consistently improve the performance on informativeness, up to 0.072 Spearman correlation on the NeR18 18 dataset
and 0.055 SummEval. However, the perfor- mance from other perspectives such as fluency and factuality do not show consistent improvements, which we will elaborate on later ($4.4.2).
Table 5: Results on Rank and QAGS datasets. where "Q" represents QAGS. Metries achieve highest correlation are bold.
Analysis on Factuality Datasets The goal of these datasets to judge whether short generated sum- mary is faithful to the original long documents. As shown in Tab. 5, we observe that (1) BARTS + CNN can almost match human baseline Rank19 and outperform all other metrics, including the most recent aop-prformmin factuality metrics FactCC and QAGS by a large margin. (2) Using paraphrase fine-tuning task will reduce BARTSCORE'S performance, which is reasonable since these two texts (i.e., the summary and document) shouldn't 't maintain the paraphrased relationship in general. (3) Introducing prompts does bring an improve- ment, even resulting in performance decrease.
Rank19
Q-CNN Q-XSUM
Acc.
Pearson
ROUGE-1
0,568
0.338
-0,008
ROUGE-2
0.630
0.459
0.097
ROUGE-L
0.587
0.357
0,024
BERTScore
0.713
0.576
0.024
MoverScore
0.713
0.414
0.054
PRISM
0.780
0.479
0.025
FactCC [30]
0.700
QAGS [66]
0,721
0,545
0.175
Human [14]
0.839
BARTSCORE
0.684
0.661
0.009
+ CNN
0.836;
0.735;
0.184;
CNN Para
0.788
0.680
0.074
+ CNN Prompt
0.796
0.7191
0.094
4.3.3 Data-to-text
The experiment results data-to-text datasets are shown in Tab. 6. We observe that (1) fine- tuning on the CNNDM dataset can consistently boost the correlation, for example, up to 0.056 gain BAGEL. (2) Additionally, further fine- tuning on paraphrase datasets results in even higher performance compared the version without any fine-tuning, up to 0.083 Spearman correlation on BAGEL dataset. These results surpass all existing top-performing metrics. (3) Our proposed prompt combination strategy can consistently improve correlation, on average 0.028 Spearman correlation. This is consistent with the findings in $4.3.2 that can improve the aspect informativeness through proper prompting.
Table 6: Results on data-to-text datasets. We report Spearman correlation. Metrics achieve highest cor- relation are bold.
BAGEL
SPRES
SFHOT
Avg.
ROUGE-1
0.234
0.115
0.118
0.1.6
ROUGE-2
0.199
0.116
0.088
0.134
ROUGE-L
0.189
0.103
0.110
0.134
BERTScore
0.289
0.156
0.135
0.193
MoverScore
0.284
0.153
0.172
0.203
PRISM
0.305
0.155
0.196
0.219
BARTSCORE
0.247
0.164
0.158
0.190
CNN
0.303
0.191
0.190
0.228
CNN Para
0.330
0.1851
0.211
0.242
f2 Prompt
0.336
0.238;
0.235;
0.270
4.4 Analysis
We design experiments to better understand the mechanism by which BARTSCORE obtains these
promising results, specifically asking three questions: Q1: Compared to other unsupervised metrics, where does BARTSCORE outperform them? Q2: How does adding prompts benefit evaluation? Q3: Will BARTScore introduce biases in unpredictable ways?

---

0.35
IBP HPR BBL. teCO IoRA
100
0.33
Kendall's Tau
0.3
Kendall's Tau
80
0.32
Percntage
0.25
0.31
40
0.2
0.3
0.15
0:29
All
15-25
25-30
35-45
45-54
SEM
LIN
FAC
(a) Top-k systems
(b) Reference length
(c) Evaluation perspective
Figure Fine-grained analysis (a,b) and prompt analysis (c). In (a. b), BE, PR, BL, CO, BA represent BERTScore, PRISM, BLEURT, COMET and BARTSCORE respectively. In (c), SEM, LIN, FAC denote semantic overlap, linguistic quality and factual correctness respectively.
4.4.1 Fine-grained Analysis
To answer QI, we choose the MT task and break down the performance of each metric into different buckets based on different axes.
Top-k Systems We report the average correlation across all language pairs achieved by each metric given only translations from top-k systems. We vary the number of k, and the results are shown in Fig. 2-(a). We can see that BARTSCORE can outperform all other metrics (including one supervised metric BLEURT) except the existing state-of-the-ant supervised metric COMET for different and the decrease in correlation becomes smoother than others when considering top-scoring systems. This indicates that BARTSCORE is robust to high-quality generated texts.
Reference Length We break down each test set into four buckets based on the reference length, which are [15, 25). [25, 35). [35, 45), [45, 54] and compute the Kendall's Tau average correlation of different metries across all language pairs within each bucket. The results are shown Fig. 2-(b). We observe that BARTSCORE can outperform or tie with other unsupervised metrics over different reference lengths. Also, its correlation with human judgments is more stable compared to all other metrics. This indicates its robustness to different input lengths. More other analyses can be found in Appendix.
4.4.2 Prompt Analysis
For Q2, we choose the summarization and data-to-text tasks for analysis where we used all prompts from prompt set. We first group all the evaluation perspectives into three categories: (I) semantic overlap (informativeness, pyramid score, and relevance) (2) linguistic quality (fluency, coherence) (3) factual correctness (factuality). We then calculate the percentage of prompts that result in performance improvements for each perspective within a dataset. Finally, we compute the average percentage of prompts that can lead to performance gains for each category. The results are shown in Tab. 2-(c). We can see that for semantic overlap, almost all prompts can lead to the performance increase, while for faciuality only few prompts improve the performance. This also explains the results in $4.3.2 where we found that combining the results of different prompts can lead to consistent increases semantic overlap but worse performance in factuality. Regarding linguistic quality, the effect of adding a prompt is not that predictive, which is also consistent with our findings in $4.3.2.
4.4.3 Bias Analysis
To answer Q3, we conduct bias analysis. Bias would indicate that the scores are too high or too low compared the scores they are given by human annotators. Therefore, see whether such biases exist, we inspected the rank differences given by human annotators and BARTScore (fine-tuned on CNNDM dataset) on the REAL Summ dataset where 24 systems are considered, including both abstractive models and extractive models as well as models based on pre-trained models and models
'In each bucket, we remove the language pairs that do not contain over 500 samples. This results in the removal of kk-en in [35, 45) and the removal of gu-en, kk-en, -t-en in [45, 54].

---

Rani Dillemnts
E9 E10
A5 NG
All A12 A13 AM
Figure 3: Bias analysis of BARTScore. The "Rank Difference" is the rank obtained using human judgements minus the rank got from BARTScore. Systems beginning with letter "E" are extractive systems while systems beginning with letter "A" abstractive systems.
that are trained from scratch. We list all the systems below. And the resulting rank difference is shown in Fig. 3.
Extractive Systems E1: BanditSum [11]: E2: Refresh [48]; E3: NeuSum [80]; E4: LSTM-
PN-RL [79]; ES: BERT-TF-SL [79]; E6: BERT-TF-PN [79]; E7: BERT-LSTM-PN-RL [79]: E8:
BERT-LSTM-PN [79]; E9: HeterGraph [68]; E10: MatchSum [78].
Abstractive Systems Al: Ptr-Gen [61]; A2: Bottom-up [17]; A3: Fast-Abs-RL [5]; A4: Two-stage-
RL [73]; A5: BERT-Ext-Abs [41]; A6: BERT-Abs [41]; A7: Trans-Abs [41]; A8: UniLM-1 [10]; A9:
UniLM-2 [2]; A10: T5-base [55]; A11: T5-large [55]; A12: T5-11B [55]; A13: BART [33]; A14:
SemSim [42].
As shown in Fig. 3, BARTScore is less effective at distinguishing the quality of extractive summariza- tion systems while much better at distinguishing However, given that there is trend for using abstractive systems as more and more pre-trained scquence-to-sequence models being proposed, BARTScore's weaknesses on extractive systems will be mitigated.
quality abstractive summarization systems.
5 Implications and Future Directions
In this paper, we proposed metric BARTSCORE that formulates evaluation of generated text as text generation task, and empirically demonstrated its efficacy. Without the supervision of human judgments, BARTSCORE
effectively evaluate texts from 7 perspectives and achieve the best performance on 16 of 22 settings against existing top-scoring metrics. We highlight potential future directions based what we have learned.
Prompt-augmented metrics As an easy-to-use but powerful method, prompting [39] has achieved impressive performance particularly semantic overlap- based evaluation perspectives, However, its effectiveness in factuality and linguistic quality- perspectives has not been fully demonstrated in this paper. In the future, more works can explore how to make better use of prompts for these and other evaluation scenarios.
Co-evolving evaluation metries and systems BARTSCORE builds the connection between metric design and system design, which allows them to share their technological advances, thereby progress- ing together. For example, a better BART-based summarization system may be directly used as more reliable automated metric for evaluating summaries, and this work makes them connected.
Acknowledgments
The authors would like to thank the anonymous reviewers for their insightful comments and sugges- tions. The authors also thank Wei Zhao for assisting with reproducing baseline results. This work was supported by the Air Force Research Laboratory under agreement number FA8750-194 111500-9-2-0200 The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory the U.S. Government.
10

---

References
[1] Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learning to retrieve reasoning paths over wikipedia graph for question answering. arXiv preprini arXiv:1911. 19440000 2019.
[2] Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao- Wuen Hon. Unilmv2: Pseudo-masker language models for unified language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of PMLR, 2020.
Proceedings of Machine Learning Research, pages
[3] Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, and Pengfei Liu. Metries also disagree in the low scoring range: Revisiting summarization evaluation metrics. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5702-5711, Barcelona, Spain (Online), December 2020, International Committee on Computational Linguistics.
[4] Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online, November 2020. Association for Computational Linguistics.
[5] Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selectere sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pages 675-686, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[6] Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[7] Hoa Trang Dang. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 12 2005.
[8] Michael Denkowski and Alon Lavie. Extending the METEOR machine translation evaluation metric to the phrase level. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Compulational Linguistics, pages 250-253, Los Angeles, California, June 2010. Association for Computational Linguistics.
[9] Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4188 2019.
[10] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao Wuen Hon. Unified language model pre-training for natural language under- standing and generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence "Alché-Buce Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042-13054, 2019.
[11] Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. Bandit- Sum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3748, Brussels, Belgium, October-November 2018 Association for Computational Linguistics.
[12] Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. GSum: A general framework for guided neural abstractive summarization. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), Mexico City, June 2021.
[13] A. R. Fabbri, Wojeiech Kryscinski. Bryan McCann, R. Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics, 9:391-409, 2021.
[14] Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Tryna Gurevych. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for 2019.
Computational Linguistics, pages

---

[15] David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York, 2007.
[16] Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098 4109, 2018.
[17] Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 4098 4109, Brussels, Belgiun, October-November 2018, Association for Computational Linguisties.
[18] Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 708-719, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[19] Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), volume 1, pages 708-719, 2018.
[20] K. Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suley- man, and P. Blunsom. Teaching machines to read and comprehend. In NIPS, 2015.
[21] Tsutomu Hirao, Manabu Okumura, Norihito Yasuda, and Hideki Isozaki. Supervised auto- matic evaluation for summarization with voted regression model. Information Processing & Management, 43(6):1521-1535 2007.
[22] J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. Large- scale, diverse, paraphrastic bitexts via sampling and clustering. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 44-54, Hong Kong, China, November 2019. Association for Computational Linguistics.
[23] Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. Scirex: A challenge dataset for document-level information extraction. arXiv preprint wxXiv:2005.00512 2020.
[24] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[25] Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972.
(26) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004. 0499000 2020.
[27] M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
[28] Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388 395, Barcelona, Spain, July 2004. Association for Computational Linguistics.
[29] Philipp Koehn. Statistical machine translation. Cambridge University Press, 2009.
[30] Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online, November 2020. Association for Computational Linguistics.
[31] Teven Le Scao and Alexander Rush. How many data points is prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627-2636, Online, June 2021. Association for Computational Linguistics.
[32] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeece pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rX1v.1910.13461, 2019.
12

---

[33] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to srueence-o-sqqueece pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020. Association for Computational Linguistics.
[34] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. ArXiv, abs/2101.001
2021.
[35] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
[36] Chin- Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co- occurrence statisties. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150-157, 2003.
[37] Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. A joint neural model for information extrac- tion with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999-8000 1999-8009, 2020.
[38] Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaicheng Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. Explainaboard: An explainable leaderboard for nlp. arXiv preprint 2XXi:2210400388 2021.
[39] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021.
[40] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint 1908 000455...4...... 2019.
[41] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lunguage Processing (EMNLP-IJCNLA) pages 3730-3740, Hong Kong, China, November 2019, Association for Computational Linguistics. [42] Yixin Liu and Pengfei Liu. Simcls: A simple framework for contrastive learning of abstractive summarization. arXiv preprint ar arXiv:2106.00 08900 2021.
[43] Qingsong Ma, Ondíej Bojar, and Yvette Graham. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671-688, Belgium, Brussels, October 2018. Association for Computational Linguistics.
[44] Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. Results of the WMT19 metries shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). pages 62-90. Florence, Italy, August 2019. Association for Computational Linguistics.
[45] François Mairesse, Milica Gasic, Filip Jurcicek, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguisties, pages 1552-1561. 2010.
[46] Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984 4997, Online, July 2020. Association for Computational Linguistics.
[47] Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't 't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807 2018.
[48] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Ranking sentences for extractive sum- marization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Pupers), pages 1747-1759, New Orleans, Louisiana, June 2018 Association for Computational Linguistics.
13

---

[49] Ani Nenkova and Rebecca Passonneau. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLTT-AACI, 2004, pages 145-152, Boston, Massachusetts, USA, May 2 May 2004. Association for Computational Linguistics.
[50] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
[51] Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74-84, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[52] Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal, Septeee September 2015. Association for Computational Linguistics.
[53] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
[54] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 1-67, 2020.
[55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020.
[56] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702, Online, November 2020. Association for Computational Linguistics.
[57] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-7, 2021.
[58] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. arXiv preprini arXiv:2001. 00666,, 2020.
[59] Timo Schick and Hinrich Schütze. Few-shot text generation with pattern-exploiting training. XrXiv preprini arXiv:20/2.1 1926, 2020.
[60] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint vrXiv:2009.0711 07118, 2020.
[61] Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[62] Thibault Sellam, Dipanjan Das. and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881 1- 1-892, Online, July 2020. Association for Computational Linguistics.
[63] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprini arXiv:2010. 101110055500 15980, 2020.
[64] Miloš Stanojević and Khalil Sima'an an. BEER: BEtter evaluation as ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414-419, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics.
[65] Brian Thompson and Matt Post. Automatic machine translation evaluation many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference Empirical Methods in Natural I Language Processing (EMNLP), pages 90-121, Online, November 2020. Association for Computational Linguistics.
14

---

[66]
Alex Wang, Kyunghyun Cho, and M. Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In ACL, 2020.
[67] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxVIPP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics.
[68] Danqing Wang, Pengfei Liu, Yining Zheng. Xipeng Qiu, and Xuanjing Huang. Heterogeneous graph neural networks for extractive document summarization. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 6209-6219, Online, July 2020. Association for Computational Linguistics.
[69] Tsung- Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. Semantically conditioned isted natural language generation for spoken dialogue systems. arXiv preprint arviv: 558.01745, 2015.
[70]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[71] Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing? arXiv preprint arXiv:2102.001166 2021.
[72] Jerrold H Zar. Spearman rank correlation. Encyclopedia of Biostatistics, 7, 2005.
[73] Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference Computational Natural Language Learning (CoNLL), pages 789-797, Hong Kong, China, November 2019. Association for Computational Linguistics.
[74] Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 1 11328-11333 1139. PMLR, 2020.
[75] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020.
[76] Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Mover- Score: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLPP pages 563-578, Hong Kong, China, November 2019. Association for Computational Linguistics.
[77] Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive summarization as text matching. arXiv preprint arXiv:2004.08795. 2020.
[78] Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1298, Online, July 2020. Association for Computational Linguistics.
[79] Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Searching for effective neural extractive summarization: What works and what's 's next. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pages 1049-1058, Florence, Italy, July 2019, Association for Computational Linguistics.
[80] Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pages Melbourne, Australia, July 2018. Association for Computational Linguistics.
15


Reviews:
[{"review_id": "pC_X4_ZFf-", "reviewer": "Reviewer_rF8J", "summary": "This paper introduces BARTScore, a new metric for generation based on the BART model. BARTScore frames the evaluation of generated text as a text generation problem. Additionally, the authors show prompts can be used to improve the metrics. BARTScore is evaluated on various generation tasks, including summarization, machine translation (MT) and data-to-text. The results are convincing: BARTScore and its variants outperform unsupervised metrics in nearly all cases and are comparable to the best supervised metrics on MT.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces BARTScore, a new metric for generation based on the BART model. BARTScore frames the evaluation of generated text as a text generation problem. In practice, this means computing conditional probabilities under a (fine-tuned) BART model. For instance, a precision score can be obtained by computing p(h|r), the probability of generating the hypothesis given a reference text. Similarly, faithfulness can be computed as p(h|s) (generating the hypothesis given the source document) and recall can be computed as p(r|h). \n\nBARTScore is unsupervised, in the sense that it does not require human judgements scores. However, fine-tuning the BART model on task-relevant datasets improves its correlation with human judgements. The authors experiment with tuning on CNN/DailyMail and tuning on CNN then ParaBank2.  This puts BARTScore in between fully rule-based metrics (e.g: n-gram overlap such as BLEU/ROUGE), and metrics directly fine-tuned to replicate human judgements (e.g BLEURT). \n\nThe authors show that adding prompts to BARTScore can further help its correlation with human performance. This is an interesting finding in itself and in line with recent findings on the effectiveness of prompting for large LMs.\n\nBARTScore is evaluated on various generation tasks, including summarization, machine translation (MT) and data-to-text. The results are convincing: BARTScore and its variants outperform unsupervised metrics in nearly all cases and are comparable to the best supervised metrics on MT.  \n\nPros:\n- BARTScore is highly correlated with human judgments in a variety of generation settings. It also does not require pre existing human judgements. In addition, significance testing is performed, a welcome addition that confirms the usefulness of BARTScores.   \n- One possible caveat is that there are many BARTScores (4 metrics * 4 models tested), and score selection can be an issue when there are no dev set human judgements available. I would like to see this mentioned, however, I do not believe this to be a dealbreaker: 1/ The metric is chosen depending on the task, not selected 2/ The BARTScore variant that performs best for each task often makes intuitive sense (fine-tuned on paraphrasing for MT, on summarization for summarization) 3/ On many tasks, several BARTScore variants are ahead of other metrics. \n- The idea of using prompting to make model-based metrics more effective is interesting, and shows great results here. I believe this is a worthwhile contribution per se. It will surely lead to other papers exploring this, as the idea can be pushed further (e.g: continuous prompts, etc.). \n- The paper is very well-written. The Preliminaries section in particular is great and should be required reading for researchers focused on generation metrics. \n- The code for BARTScore is already published, with a clear README. There is also a leaderboard for metric evaluation. Both of those will be very valuable to the community. \n- The analysis, though short, has interesting findings. \n\nCons:\n- It would be great to see some analysis on the impact of model choice to understand the generality of BARTScore. In particular, I am curious about:\n  1.  How well does this work if BART is replaced with another, worse-performing model? Using older models to evaluate the quality of more recent ones can lead to issues as shown in BLEURT. I do not believe 4.4.1 fully addresses this given many evaluated models are either BART or older. \n  2.  Are the scores improved on some tasks due to the base metric model being the same as the models evaluated (e.g Factuality evaluation of BART based outputs)?\n\n~~~~ \n\nAuthor response edit: I still feel like this paper deserves the high grade I assigned it. Here is my rationale for sticking with a higher grade than my fellow reviewers:\n\n- Many of the concerns highlighted by my fellow reviewers are interesting research questions themselves that apply more generally to learned/co-evolved evaluation methods (how much does it help to use a metric based on the same model being tested? tuned in the same way? trained in a seq-to-seq vs BERT like setup? what new biases do learned metrics introduce? What is the role of human evals in the co-evolution of metrics and models?). Although this paper does not answer all these questions satisfactorily, this cannot be expected of a single paper and the code release and result repository will prove fruitful for such studies. \n\n- The prompt idea is very interesting in itself, something which has not been very emphasized in other reviews (aside from bvYP).\n\nConsequently, and after reading the other reviews and response, I have increased my confidence score.\n\nThe main recommendation I have is that the authors mention the metric selection issue more explicitly (Weakness 1 of Rev TKFn, which I agree at least partially with cf my caveat). \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Do not think they are issues here. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "KjCVPgwobj-", "reviewer": "Reviewer_TKFn", "summary": "This paper proposes BARTScore, a method to evaluate NLG system outputs by using a NLG system's output likelihood score, in particular different variants of BART. By changing what BARTScore conditions on and what it must score, as well as the prompts which are fed to the BARTScore model, variants of the evaluation method can be derived (faithfulness, precision, recall, F), and be adapted to measure different generation qualities such as coverage, coherence, factuality, fluency, informativeness, and adequacy. Experiments are run on a number of NLG tasks including MT, summarization, factuality, and data-to-text, where higher correlations are found between versions of BARTScore and human judgments. There are also several extra analytical experiments on MT focusing on issues such as reference length and the choice of prompts.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a new method to evaluate NLG system outputs based on using scores derived from finetuned BART-based conditional language models. There are a large number of experiments to evaluate the correlation of the method to human judgments. The paper is clearly written and easy to understand, and the contribution should be significant to the NLG community if it is adopted.\n\nStrengths:\n - The proposed method is simple and flexible. I appreciated the different methods of applying BARTScore that consider the relation between the source text, the reference output, and the reference output, which can be adapted to the quality of the text being evaluated.\n - BARTScore achieves good correlations on a range of NLG datasets in a range of NLG tasks.\n\nNevertheless, there are a few issues with the paper which are concerning.\n\nWeaknesses:\n1 - First, the reporting of the results and the comparisons to previous methods are unfair and potentially misleading. For some of the results, notably Table 4 for summarization evaluation, BartScore gets an advantage in being able to select different variants of itself for the different subtasks, as described by the paper. This is in comparison to other methods which must stick to one variant, such as ROUGE-1 or ROUGE-L. Second, many of the previous measures presented are not designed to measure the stated qualities. For example, ROUGE-1 was never meant to measure coherence, or indeed factuality, or fluency. Thus the table is misleading and overstates the advantage that BARTScore variants have over previous methods.\n \n2 -  On a different level, even if we take the better correlations at face value, I am concerned that BARTScore would introduce biases in unpredictable ways. For example, could it privilege models that are more BART-like over NLG models that are not based on pre-trained LMs? I would be interested in seeing whether the correlations hold within AND between model classes. e.g., what if they were applied to a dataset with a combination of neural and pre-neural MT or summarization systems? What if BARTScore were applied to reference outputs themselves?\n\n3 - Another interesting analysis to perform would be to examine and evaluate the output of decoding directly from the probability distribution of BARTScore. I would expect BARTScore to break down when evaluating these extrema or similar-looking sentences, and a question to ask is how do we know if we encounter such cases?\n\n---\nEdit:\nThe response and the proposed revisions addressed Weaknesses 2 and 3 above satisfactorily, but we still disagree regarding Weakness 1. I've updated my score to reflect this.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There isn't any deep discussion of limitations or societal impacts of their work. There is some discussion of how a research community that adopts BARTScore might lead to a research cycle where the evaluation method and the system would co-evolve. It is unclear to me whether this would be a positive development.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "1M6tu4spLz2", "reviewer": "Reviewer_3xwH", "summary": "This paper tries to propose a new generation-based BARTScore to evaluate text generation problems. The authors want to fill the gap between pretrainining objectives and the down stream feature extractors.  The main idea is that BART can achieve better score when the generation results are better. The BART Score is an unsupervised metric and have better correlation with different perspectives. The paper also check the influence of different textual prompts and fine-tuning process. The paper conducts experiments from 7 perspectives on 16 datasets. ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper gives a pretty solid literature survey. The paper first formally defined gold-standard human evaluation with 7 different perspectives. The paper then characterizes differences in task formulation and evaluation perspectives. The baselines are all state-of-the-art or widely used in the generation/summarization/translation tasks. The paper also provides code and sample scripts for users to check. The paper also provides pre-trained model selection and prompt set in the supplements. \n\n2. The idea to use seq2seq generation tasks is pretty novel. The paper provides an unsupervised way to evaluate generation results with the transformer-based pre-trained seq2seq model. The paper provides four ways to apply their model according to different situations. The experiments are pretty comprehensive. The paper compares BARTscore with supervised and unsupervised metrics by using multiple evaluations. The paper also provides insightful analysis about reference length and differences prompt. \n\nWeaknesses:\n1. In the experiments for translation Table 3 and Figure 2, I don't see a clear difference between COMET and BARTSCORE. The COMET has close or higher scores compared to BARTSCORE. It would be better for authors to clarify the detailed advantages in the paper. The authors seem to only focus on the automatic metrics with quantitative evaluation. I would like to see more qualitative evaluation with detailed examples like section 4.4.2.\n\n2. In the paper, it is not clear the specific BART model. The paper might be better to state that they use BART-large-CNN as shown in the code. Also, I'm curious about the reason behind the bad performance of T5 and PEGASUS. Moreover, it seems that T5-base-CNN only has about 200M parameters. However, its performance is comparable with BART-CNN which has 400M parameters. I'm wondering why authors choose BART-CNN instead of T5-base-cnn. The paper also fails to specify which subset of prompt they uses for each task. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper describes some limitations in section 5. The paper also provides future work directions in that section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1Hh5WZwfh_O", "reviewer": "Reviewer_bvYP", "summary": "This paper proposes a different paradigm of evaluation of text generation by modeling evaluation as a text generation problem. This is built on top of the BART text generation algorithm, hence, the metric is called BARTScore. Similar to BERTScore, which is based on BERT, the metric proposed in this paper requires no training (with the reference human judgments) as the pre-trained BART model is used. BARTScore considers various aspects of matching generated text with reference (like factuality, coherence, etc.). In addition, BARTScore is shown to improve with using textual prompts and with fine-tuning on downstream domain tasks. BARTScore shows promising results across the board and fine-grained analysis shows effectiveness across different perspectives.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points:\n1. The paper is well-motivated and clear. The idea of a different paradigm for evaluation is interesting.\n2. The premise of the idea is well-positioned against relevant prior work.\n3. An important contribution is the addition of prompts which can lead to better evaluations.\n4. A bunch of different standard metrics has been used to compare BARTScore with promising results.\n5. The fine-grained analysis in Sec 4.4 offers some necessary insights.\n\n## Questions:\n1. The idea of using text generation as an evaluation of text generation is not entirely new (as the authors themselves compared with PRISM). So, the BARTScore formulation and the finer points of differences with PRISM seem incremental.\n2. It is not clear why the authors resorted to weighing each token equally (is it experimentally validated against IDF?).\n3. There is a concern for this kind of metric which is dependent upon existing text generation like BART. The paper also talks about trying out T5 and PEGASUS. The metric which is dependent upon a pre-trained model is like a moving target with no clear grounding of evaluations. Let's say a future model BART++ comes up, how do you decide whether it is better to have BARTScore or BART++Score. It's also not clear how to establish that BART++ is better than BART in text generation when the evaluation metrics are moving targets. The authors advocated this paradigm of evaluation and text-gen systems co-evolving together but that can introduce significant biases without the proper grounding of evaluations.\n4. Fine-tuning on downstream tasks is not surprising and is a positive outcome. However, the concern remains because the evaluation is now dependent on the dataset being fine-tuned on, which aggravates the need for having a non-moving gold standard evaluation as highlighted above.\n5. Section 4.1.2 definitions seem lifted from Wikipedia word-by-word even though some other articles have been cited.\n6. The paper gives a feel of being hurriedly written, and a lot of work is needed to make it a coherent and fun read.\n7. A bunch of different datasets has been considered for comparison covering various aspects like coherence, consistency, informativeness, etc. However, for Factuality and Data-to-text datasets, the number of samples for evaluation is in 100s, the scale of which is very less to lead to a strong conclusion about the effectiveness of BARTScore over others.\n8. In Sec 4.2.2, the subsection \"Selection of BARTScore Usage\", there are intuitions about the preference of the kind of BARTScore used. Have they been experimentally validated as well?\n9. Typo: Line 159: 'encouraged' --> 'encourage'.\n\n## Originality: 5/10\n\n## Quality: 4/10\n\n## Clarity: 6/10\n\n## Significance: 4/10\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the authors have not addressed the concerns of moving targets in evaluation which can lead to significance biases in text generation systems.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for evaluation of text generation (BARTScore) by modeling evaluation as a text generation problem, built on top of the BART text generation algorithm. Similar to BERTScore, which is based on BERT, the metric proposed in this paper requires no training. BARTScore is also shown to improve with using textual prompts and with fine-tuning on downstream domain tasks, showing good results across the board for evaluating various generation tasks including MT and summarization, approaching evaluation metrics such as COMET which are supervised on human assessments. 

All reviewers feel positively about this paper. While the underlying idea is not particularly creative, this is a solid paper which proposes a simple and effective approach for evaluation of text generation with convincing results and which is likely to be impactful.
Decision: Accept (Poster)