Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a different paradigm of evaluation of text generation by modeling evaluation as a text generation problem. This is built on top of the BART text generation algorithm, hence, the metric is called BARTScore. Similar to BERTScore, which is based on BERT, the metric proposed in this paper requires no training (with the reference human judgments) as the pre-trained BART model is used. BARTScore considers various aspects of matching generated text with reference (like factuality, coherence, etc.). In addition, BARTScore is shown to improve with using textual prompts and with fine-tuning on downstream domain tasks. BARTScore shows promising results across the board and fine-grained analysis shows effectiveness across different perspectives.

main_review: ## Strong points:
1. The paper is well-motivated and clear. The idea of a different paradigm for evaluation is interesting.
2. The premise of the idea is well-positioned against relevant prior work.
3. An important contribution is the addition of prompts which can lead to better evaluations.
4. A bunch of different standard metrics has been used to compare BARTScore with promising results.
5. The fine-grained analysis in Sec 4.4 offers some necessary insights.

## Questions:
1. The idea of using text generation as an evaluation of text generation is not entirely new (as the authors themselves compared with PRISM). So, the BARTScore formulation and the finer points of differences with PRISM seem incremental.
2. It is not clear why the authors resorted to weighing each token equally (is it experimentally validated against IDF?).
3. There is a concern for this kind of metric which is dependent upon existing text generation like BART. The paper also talks about trying out T5 and PEGASUS. The metric which is dependent upon a pre-trained model is like a moving target with no clear grounding of evaluations. Let's say a future model BART++ comes up, how do you decide whether it is better to have BARTScore or BART++Score. It's also not clear how to establish that BART++ is better than BART in text generation when the evaluation metrics are moving targets. The authors advocated this paradigm of evaluation and text-gen systems co-evolving together but that can introduce significant biases without the proper grounding of evaluations.
4. Fine-tuning on downstream tasks is not surprising and is a positive outcome. However, the concern remains because the evaluation is now dependent on the dataset being fine-tuned on, which aggravates the need for having a non-moving gold standard evaluation as highlighted above.
5. Section 4.1.2 definitions seem lifted from Wikipedia word-by-word even though some other articles have been cited.
6. The paper gives a feel of being hurriedly written, and a lot of work is needed to make it a coherent and fun read.
7. A bunch of different datasets has been considered for comparison covering various aspects like coherence, consistency, informativeness, etc. However, for Factuality and Data-to-text datasets, the number of samples for evaluation is in 100s, the scale of which is very less to lead to a strong conclusion about the effectiveness of BARTScore over others.
8. In Sec 4.2.2, the subsection "Selection of BARTScore Usage", there are intuitions about the preference of the kind of BARTScore used. Have they been experimentally validated as well?
9. Typo: Line 159: 'encouraged' --> 'encourage'.

## Originality: 5/10

## Quality: 4/10

## Clarity: 6/10

## Significance: 4/10


limitations_and_societal_impact: No, the authors have not addressed the concerns of moving targets in evaluation which can lead to significance biases in text generation systems.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper tries to propose a new generation-based BARTScore to evaluate text generation problems. The authors want to fill the gap between pretrainining objectives and the down stream feature extractors.  The main idea is that BART can achieve better score when the generation results are better. The BART Score is an unsupervised metric and have better correlation with different perspectives. The paper also check the influence of different textual prompts and fine-tuning process. The paper conducts experiments from 7 perspectives on 16 datasets. 
main_review: Strengths:
1. The paper gives a pretty solid literature survey. The paper first formally defined gold-standard human evaluation with 7 different perspectives. The paper then characterizes differences in task formulation and evaluation perspectives. The baselines are all state-of-the-art or widely used in the generation/summarization/translation tasks. The paper also provides code and sample scripts for users to check. The paper also provides pre-trained model selection and prompt set in the supplements. 

2. The idea to use seq2seq generation tasks is pretty novel. The paper provides an unsupervised way to evaluate generation results with the transformer-based pre-trained seq2seq model. The paper provides four ways to apply their model according to different situations. The experiments are pretty comprehensive. The paper compares BARTscore with supervised and unsupervised metrics by using multiple evaluations. The paper also provides insightful analysis about reference length and differences prompt. 

Weaknesses:
1. In the experiments for translation Table 3 and Figure 2, I don't see a clear difference between COMET and BARTSCORE. The COMET has close or higher scores compared to BARTSCORE. It would be better for authors to clarify the detailed advantages in the paper. The authors seem to only focus on the automatic metrics with quantitative evaluation. I would like to see more qualitative evaluation with detailed examples like section 4.4.2.

2. In the paper, it is not clear the specific BART model. The paper might be better to state that they use BART-large-CNN as shown in the code. Also, I'm curious about the reason behind the bad performance of T5 and PEGASUS. Moreover, it seems that T5-base-CNN only has about 200M parameters. However, its performance is comparable with BART-CNN which has 400M parameters. I'm wondering why authors choose BART-CNN instead of T5-base-cnn. The paper also fails to specify which subset of prompt they uses for each task. 
limitations_and_societal_impact: The paper describes some limitations in section 5. The paper also provides future work directions in that section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes BARTScore, a method to evaluate NLG system outputs by using a NLG system's output likelihood score, in particular different variants of BART. By changing what BARTScore conditions on and what it must score, as well as the prompts which are fed to the BARTScore model, variants of the evaluation method can be derived (faithfulness, precision, recall, F), and be adapted to measure different generation qualities such as coverage, coherence, factuality, fluency, informativeness, and adequacy. Experiments are run on a number of NLG tasks including MT, summarization, factuality, and data-to-text, where higher correlations are found between versions of BARTScore and human judgments. There are also several extra analytical experiments on MT focusing on issues such as reference length and the choice of prompts.
main_review: This paper proposes a new method to evaluate NLG system outputs based on using scores derived from finetuned BART-based conditional language models. There are a large number of experiments to evaluate the correlation of the method to human judgments. The paper is clearly written and easy to understand, and the contribution should be significant to the NLG community if it is adopted.

Strengths:
 - The proposed method is simple and flexible. I appreciated the different methods of applying BARTScore that consider the relation between the source text, the reference output, and the reference output, which can be adapted to the quality of the text being evaluated.
 - BARTScore achieves good correlations on a range of NLG datasets in a range of NLG tasks.

Nevertheless, there are a few issues with the paper which are concerning.

Weaknesses:
1 - First, the reporting of the results and the comparisons to previous methods are unfair and potentially misleading. For some of the results, notably Table 4 for summarization evaluation, BartScore gets an advantage in being able to select different variants of itself for the different subtasks, as described by the paper. This is in comparison to other methods which must stick to one variant, such as ROUGE-1 or ROUGE-L. Second, many of the previous measures presented are not designed to measure the stated qualities. For example, ROUGE-1 was never meant to measure coherence, or indeed factuality, or fluency. Thus the table is misleading and overstates the advantage that BARTScore variants have over previous methods.
 
2 -  On a different level, even if we take the better correlations at face value, I am concerned that BARTScore would introduce biases in unpredictable ways. For example, could it privilege models that are more BART-like over NLG models that are not based on pre-trained LMs? I would be interested in seeing whether the correlations hold within AND between model classes. e.g., what if they were applied to a dataset with a combination of neural and pre-neural MT or summarization systems? What if BARTScore were applied to reference outputs themselves?

3 - Another interesting analysis to perform would be to examine and evaluate the output of decoding directly from the probability distribution of BARTScore. I would expect BARTScore to break down when evaluating these extrema or similar-looking sentences, and a question to ask is how do we know if we encounter such cases?

---
Edit:
The response and the proposed revisions addressed Weaknesses 2 and 3 above satisfactorily, but we still disagree regarding Weakness 1. I've updated my score to reflect this.

limitations_and_societal_impact: There isn't any deep discussion of limitations or societal impacts of their work. There is some discussion of how a research community that adopts BARTScore might lead to a research cycle where the evaluation method and the system would co-evolve. It is unclear to me whether this would be a positive development.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces BARTScore, a new metric for generation based on the BART model. BARTScore frames the evaluation of generated text as a text generation problem. Additionally, the authors show prompts can be used to improve the metrics. BARTScore is evaluated on various generation tasks, including summarization, machine translation (MT) and data-to-text. The results are convincing: BARTScore and its variants outperform unsupervised metrics in nearly all cases and are comparable to the best supervised metrics on MT.  
main_review: This paper introduces BARTScore, a new metric for generation based on the BART model. BARTScore frames the evaluation of generated text as a text generation problem. In practice, this means computing conditional probabilities under a (fine-tuned) BART model. For instance, a precision score can be obtained by computing p(h|r), the probability of generating the hypothesis given a reference text. Similarly, faithfulness can be computed as p(h|s) (generating the hypothesis given the source document) and recall can be computed as p(r|h). 

BARTScore is unsupervised, in the sense that it does not require human judgements scores. However, fine-tuning the BART model on task-relevant datasets improves its correlation with human judgements. The authors experiment with tuning on CNN/DailyMail and tuning on CNN then ParaBank2.  This puts BARTScore in between fully rule-based metrics (e.g: n-gram overlap such as BLEU/ROUGE), and metrics directly fine-tuned to replicate human judgements (e.g BLEURT). 

The authors show that adding prompts to BARTScore can further help its correlation with human performance. This is an interesting finding in itself and in line with recent findings on the effectiveness of prompting for large LMs.

BARTScore is evaluated on various generation tasks, including summarization, machine translation (MT) and data-to-text. The results are convincing: BARTScore and its variants outperform unsupervised metrics in nearly all cases and are comparable to the best supervised metrics on MT.  

Pros:
- BARTScore is highly correlated with human judgments in a variety of generation settings. It also does not require pre existing human judgements. In addition, significance testing is performed, a welcome addition that confirms the usefulness of BARTScores.   
- One possible caveat is that there are many BARTScores (4 metrics * 4 models tested), and score selection can be an issue when there are no dev set human judgements available. I would like to see this mentioned, however, I do not believe this to be a dealbreaker: 1/ The metric is chosen depending on the task, not selected 2/ The BARTScore variant that performs best for each task often makes intuitive sense (fine-tuned on paraphrasing for MT, on summarization for summarization) 3/ On many tasks, several BARTScore variants are ahead of other metrics. 
- The idea of using prompting to make model-based metrics more effective is interesting, and shows great results here. I believe this is a worthwhile contribution per se. It will surely lead to other papers exploring this, as the idea can be pushed further (e.g: continuous prompts, etc.). 
- The paper is very well-written. The Preliminaries section in particular is great and should be required reading for researchers focused on generation metrics. 
- The code for BARTScore is already published, with a clear README. There is also a leaderboard for metric evaluation. Both of those will be very valuable to the community. 
- The analysis, though short, has interesting findings. 

Cons:
- It would be great to see some analysis on the impact of model choice to understand the generality of BARTScore. In particular, I am curious about:
  1.  How well does this work if BART is replaced with another, worse-performing model? Using older models to evaluate the quality of more recent ones can lead to issues as shown in BLEURT. I do not believe 4.4.1 fully addresses this given many evaluated models are either BART or older. 
  2.  Are the scores improved on some tasks due to the base metric model being the same as the models evaluated (e.g Factuality evaluation of BART based outputs)?

~~~~ 

Author response edit: I still feel like this paper deserves the high grade I assigned it. Here is my rationale for sticking with a higher grade than my fellow reviewers:

- Many of the concerns highlighted by my fellow reviewers are interesting research questions themselves that apply more generally to learned/co-evolved evaluation methods (how much does it help to use a metric based on the same model being tested? tuned in the same way? trained in a seq-to-seq vs BERT like setup? what new biases do learned metrics introduce? What is the role of human evals in the co-evolution of metrics and models?). Although this paper does not answer all these questions satisfactorily, this cannot be expected of a single paper and the code release and result repository will prove fruitful for such studies. 

- The prompt idea is very interesting in itself, something which has not been very emphasized in other reviews (aside from bvYP).

Consequently, and after reading the other reviews and response, I have increased my confidence score.

The main recommendation I have is that the authors mention the metric selection issue more explicitly (Weakness 1 of Rev TKFn, which I agree at least partially with cf my caveat). 
 
limitations_and_societal_impact: Do not think they are issues here.
review_title: null

Meta-review: This paper proposes a framework for evaluation of text generation (BARTScore) by modeling evaluation as a text generation problem, built on top of the BART text generation algorithm. Similar to BERTScore, which is based on BERT, the metric proposed in this paper requires no training. BARTScore is also shown to improve with using textual prompts and with fine-tuning on downstream domain tasks, showing good results across the board for evaluating various generation tasks including MT and summarization, approaching evaluation metrics such as COMET which are supervised on human assessments. 

All reviewers feel positively about this paper. While the underlying idea is not particularly creative, this is a solid paper which proposes a simple and effective approach for evaluation of text generation with convincing results and which is likely to be impactful.
Decision: Accept (Poster)