BARTScore: Evaluating Generated Text as Text Generation

A wide variety of NLP applications , such as machine translation , summarization , and dialog , involve text generation . One major challenge for these applications is how to evaluate whether such generated texts are actually fluent , accurate , or effective . In this work , we conceptualize the evaluation of generated text as a text generation problem , modeled using pre-trained sequence-to-sequence models . The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better . We operationalize this idea using BART [ 32 ] , an encoder-decoder based pre-trained model , and propose a metric BARTSCORE with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives ( e.g . informativeness , fluency , or factuality ) . BARTSCORE is conceptually simple and empirically effective . It can outperform existing top-scoring metrics in 16 of 22 test settings , covering evaluation of 16 datasets ( e.g. , machine translation , text summarization ) and 7 different perspectives ( e.g. , informativeness , factuality ) . Code to calculate BARTScore is available at https : //github.com/neulab/BARTScore , and we have released an interactive leaderboard for meta-evaluation at http : //explainaboard.nlpedia.ai/leaderboard/task-meval/ on the EXPLAINABOARD platform [ 38 ] , which allows us to interactively understand the strengths , weaknesses , and complementarity of each metric . 1 Introduction . One defining feature of recent NLP models is the use of neural representations trained on raw text , using unsupervised objectives such as language modeling [ 6,53 ] , or denoising autoencoding [ 9,32,54 ] . By learning to predict the words or sentences in natural text , these models simultaneously learn to extract features that not only benefit mainstream NLP tasks such as information extraction [ 23 , 37 ] , question answering [ 1,26 ] , text summarization [ 40,77 ] but also have proven effective in development of automatic metrics for evaluation of text generation itself [ 62 , 65 ] . For example , BERTScore [ 75 ] and MoverScore [ 76 ] take features extracted by BERT [ 9 ] and apply unsupervised matching functions to compare system outputs against references . Other works build supervised frameworks that use the extracted features to learn to rank [ 56 ] or regress [ 62 ] to human evaluation scores . However , in the context of generation evaluation , one may note that there is a decided disconnect between how models are pre-trained using text generation objectives and how they are used as down-stream feature extractors . This leads to potential under-utilization of the pre-trained model parameters . For example , the output prediction layer is not used at all in this case . This disconnect is particularly striking because of the close connection between the pre-training objectives and the generation tasks we want to evaluate . ∗ Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we instead argue for a formulation of evaluation of generated text as a text generation problem , directly evaluating text through the lens of its probability of being generated from or generating other textual inputs and outputs . This is a better match with the underlying pre-training tasks and allows us to more fully take advantage of the parameters learned during the pre-training phase . We solve the modeling problem with a pre-trained sequence-to-sequence ( seq2seq ) model , specifically BART [ 32 ] , and devise a metric named BARTSCORE , which has the following characteristics : ( 1 ) BARTSCORE is parameter- and data-efficient . Architecturally there are no extra parameters beyond those used in pre-training itself , and it is an unsupervised metric that doesn ’ t require human judgments to train . ( 2 ) BARTSCORE can better support evaluation of generated text from different perspectives ( e.g. , informativeness , coherence , factuality , §4 ) by adjusting the inputs and outputs of the conditional text generation problem , as we demonstrate in §3.2 . This is in contrast to most previous work , which mostly examines correlation of the devised metrics with output quality from a limited number of perspectives . ( 3 ) BARTSCORE can be further enhanced by ( i ) providing textual prompts that bring the evaluation task closer to the pre-training task , or ( ii ) updating the underlying model by fine-tuning BART based on downstream generation tasks ( e.g. , text summarization ) . Experimentally , we evaluate different variants of BARTSCORE from 7 perspectives on 16 datasets . BARTSCORE achieves the best performance in 16 of 22 test settings against existing top-scoring metrics . Empirical results also show the effectiveness of the prompting strategy supported by BARTSCORE . For example , simply adding the phrase “ such as ” to the translated text when using BARTSCORE can lead to a 3 % point absolute improvement in correlation on “ German-English ” machine translation ( MT ) evaluation . Additional analysis shows that BARTSCORE is more robust when dealing with high-quality texts generated by top-performing systems . 2 Preliminaries . 2.1 Problem Formulation . As stated above , our goal is to assess the quality of generated text [ 3 , 46 ] . In this work , we focus on conditional text generation ( e.g. , machine translation ) , where the goal is to generate a hypothesis ( h = h1 , · · · , hm ) based on a given source text ( s = s1 , · · · , sn ) . Commonly , one or multiple human-created references ( r = r1 , · · · , rl ) are provided to aid this evaluation . 2.2 Gold-standard Human Evaluation . In general , the gold-standard method for evaluating such texts is still human evaluation , where human annotators assess the generated texts ’ quality . This evaluation can be done from perspectives , and we list a few common varieties below ( all are investigated in §4 ) : 1 . Informativeness ( INFO ) : How well the generated hypothesis captures the key ideas of the source text [ 18 ] . 2 . Relevance ( REL ) : How consistent the generated hypothesis is with respect to the source text [ 19 ] . 3 . Fluency ( FLU ) : Whether the text has no formatting problems , capitalization errors or obviously ungrammatical sentences ( e.g. , fragments , missing components ) that make the text difficult to read [ 13 ] . 4 . Coherence ( COH ) : Whether the text builds from sentence to sentence to a coherent body of information about a topic [ 7 ] . 5 . Factuality ( FAC ) : Whether the generated hypothesis contains only statements entailed by the source text [ 30 ] . 6 . Semantic Coverage ( COV ) : How many semantic content units from reference texts are covered by the generated hypothesis [ 49 ] . 7 . Adequacy ( ADE ) : Whether the output conveys the same meaning as the input sentence , and none of the message is lost , added , or distorted [ 29 ] . Most existing evaluation metrics were designed to cover a small subset of these perspectives . For example , BLEU [ 50 ] aims to capture the adequacy and fluency of translations , while ROUGE [ 36 ] was designed to match the semantic coverage metric . Some metrics , particularly trainable ones , can perform evaluation from different perspectives but generally require maximizing correlation with each type of judgment separately [ 8 ] . h1 h2r1 r2 r3 r1 r2 r3 h1 h2 s1 s2 r1 r2 r3 h1 h2 r1 r2 r3 h1 h2 s1 s2 r1 r2 r3 h1 h2 s1 s2 s r h Coherence / Fluency Factuality Relevance InformativenessConsistency ( a ) Matching h1 h2r1 r2 r3 r1 r2 r3 h1 h2 s1 s2 r1 r2 r3 h1 h2 r1 r2 r3 h1 h2 s1 s2 r1 r2 r3 h1 h2 s1 s2 s r h Coherence / Fluency Factuality Relevance InformativenessConsistency ( b ) Regression h1 h2r1 r2 r3 r1 r2 r3 h1 h2 s1 s2 r1 r2 r3 h1 h2 r1 r2 r3 h1 h2 s1 s2 r1 r2 r3 h1 h2 s1 s2 s r h Coherence / Fluency Factuality Relevance InformativenessConsistency ( c ) Ranking h1 h2r1 r r3 r1 r2 r3 1 h2 s1 s2 r1 r2 r3 h1 h2 r1 r2 r3 h1 h2 s1 s2 r1 r2 r3 h1 h2 s1 s2 s r h Coherence / Fluency Factuality Relevance InformativenessConsistency ( d ) Generation Figure 1 : Evaluation metrics as different tasks , where si , hi and rj represent source , hypothesis and reference words respectively . As we describe more in §4 , BARTSCORE can evaluate text from the great majority of these perspectives , significantly expanding its applicability compared to these metrics . 2.3 Evaluation as Different Tasks . There is a recent trend that leverages neural models for automated evaluation in different ways , as shown in Fig . 1 . We first elaborate on their characteristics by highlighting differences in task formulation and evaluation perspectives . T1 : Unsupervised Matching . Unsupervised matching metrics aim to measure the semantic equivalence between the reference and hypothesis by using a token-level matching functions in distributed representation space , such as BERTScore [ 75 ] , MoverScore [ 76 ] or discrete string space like ROUGE [ 35 ] , BLEU [ 50 ] , CHRF [ 52 ] . Although similar matching functions can be used to assess the quality beyond semantic equivalence ( e.g , factuality , a relationship between source text and hypothesis ) , to our knowledge prior research has not attested to the capability of unsupervised matching methods in this regard ; we explore this further in our experiments ( Tab . 5 ) . T2 : Supervised Regression . Regression-based models introduce a parameterized regression layer , which would be learned in a supervised fashion to accurately predict human judgments . Examples include recent metrics BLEURT [ 62 ] , COMET [ 56 ] and traditional metrics like S3 [ 51 ] , VRM [ 21 ] . T3 : Supervised Ranking . Evaluation can also be conceived as a ranking problem , where the main idea is to learn a scoring function that assigns a higher score to better hypotheses than to worse ones . Examples include COMET [ 56 ] and BEER [ 64 ] , where COMET focuses the machine translation task and relies on human judgments to tune parameters in ranking or regression layers , and BEER combines many simple features in a tunable linear model of MT evaluation metrics . T4 : Text Generation . In this work , we formulate evaluating generated text as a text generation task from pre-trained language models . The basic idea is that a high-quality hypothesis will be easily generated based on source or reference text or vice-versa . This has not been covered as extensively in previous work , with one notable exception being PRISM [ 65 ] . Our work differs from PRISM in several ways : ( i ) PRISM formulates evaluation as a paraphrasing task , whose definition that two texts are with the same meaning limits its applicable scenarios , like factuality evaluation in text summarization that takes source documents and generated summaries as input whose semantic space are different . ( ii ) PRISM trained a model from scratch on parallel data while BARTSCORE is based on open-sourced pre-trained seq2seq models . ( iii ) BARTSCORE supports prompt-based learning [ 59 , 63 ] which hasn ’ t been examined in PRISM .

Summary: This paper proposes BARTScore, a method to evaluate NLG system outputs by using a NLG system's output likelihood score, in particular different variants of BART. By changing what BARTScore conditions on and what it must score, as well as the prompts which are fed to the BARTScore model, variants of the evaluation method can be derived (faithfulness, precision, recall, F), and be adapted to measure different generation qualities such as coverage, coherence, factuality, fluency, informativeness, and adequacy. Experiments are run on a number of NLG tasks including MT, summarization, factuality, and data-to-text, where higher correlations are found between versions of BARTScore and human judgments. There are also several extra analytical experiments on MT focusing on issues such as reference length and the choice of prompts.