Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Paper Contributions

This paper introduces a new text generation scoring approach using BERT, called BERTScore. Using BERT embeddings and optionally idf scores, a greedy matching is performed between all reference and candidate words, with cosine similarity between vector representations as the scoring. From this, a precision, recall and F1 score can be derived. This notably outperforms BLEU, as well as other metrics, most but not all of the time. The paper offers a broad range of comparisons and analysis.

Decision

I'm leaning towards accepting the paper on the basis of the following.

Strong points taken in consideration:
- Simple, well-motivated metric that uses powerful BERT-style models, without being slow to compute either.
- Good performance empirically on WMT. I'm less convinced on COCO since using the image is fair game there.
- Code is provided, and it is simple and adaptable for future work.
- Experimentation is detailed and reproducible.

Weaker points taken in consideration:
- Work conducted in parallel matches or exceeds the performance of BERTScore. This shouldn't necessarily be a reason to choose not to publish this work in my opinion, but it should be taken into consideration. I like that the authors were open and clear regarding this in their discussion.
- The authors haven't come up with a recommendation for a single configuration of their approach. In one place they recommend F-BERT without idf, in another they argue for picking and choosing based on context, with little help about how to choose. I think practitioners are only going to be willing to switch away from BLEU, for example, if a single one-size-fits-all metric is proposed instead. I identify this ambiguity between BERTScore versions as an important weakness of the paper.
- It's unclear throughout whether words or wordpieces are the main token being considered. Most discussion and definitions use "words", but in section 3, subsection Token Representation, it appears to be clearly stated that BERTScore uses a BERT model based on word pieces. I recommend adjust the language to be more consistent throughout. Also, scoring examples with word pieces would be more consistent with this as well, imo. Notably, I'm actually unsure whether you compute IDF over words or word pieces, and how this is applied. 
- Finally, I found some weaknesses in the Importance Weighting section (though this isn't too important since IDF isn't part of the recommended BERTScore I believe). The IDF scores would be stronger if they were computed on a bigger in-domain corpus than the gold test set. This would add extra steps to using BERTScore though and make things more complicated in practice, but this should nevertheless probably be tried, or at least discussed in the paper. Also, the plus-one smoothing handles unknown words (or word piece?) and I'm not sure why. If we're using the test set to compute IDF, and the sentences we're looking *are* in the test set, then there shouldn't be unknown words and no smoothing is required.

So overall, I still think this deserves publication because it's valuable information for researchers, and the metric itself could be immediately useful to some as well. However, the weaknesses mentioned make me hesitate to fully endorse the work.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents a simple application of BERT-based contextual embeddings for evaluation of text generation models such as machine translation and image caption generation. An extensive set of experiments have been carried out to show that the proposed BERTScore metric achieves better correlation with human judgments than the existing metrics. Overall, the paper is well-written and the motivations are clear. However, I am not sure about the technical novelty of the paper as the proposed approach is a natural application of BERT along with traditional cosine similarity measures and precision, recall, F1-based computations, and simple IDF-based importance weighting. 

Other comments:

- It would be interesting to see how the proposed metric performs to evaluate paraphrase generation and text simplification models as the models need to follow specific constraints such as semantic equivalence, novelty, simplicity etc. with respect to source and reference sentences.

- Another limitation of the proposed metric is memory and time complexity as it takes relatively more time to evaluate the sentences compared to BLEU, as authors acknowledged in Section 5. 
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: 
*** Update ***
I'd like to thank the authors for answering my questions, and I am satisfied with their response. I have read the other reviews for this paper as well, and I am keeping my score.


This paper proposes BERTScore, a method for automatic evaluation of text. Their method uses BERT to produce contextualized word representations for the words in the reference and hypothesis. Then they compute the precision, recall, and F1 by greedily matching up words between the hypothesis and reference. To be more specific, for say recall they take each word in the reference and compute the cosine with all words in the hypothesis. Then they add up the largest cosine similarity for each word and average them together. Precision is defined similarly but with the roles of hypothesis and reference switched. F1 is then the harmonic mean of these two scores. They also experiment with using idf to weight importance.

Their method is simple, but achieves very strong results and there are a ton of experiments in this paper (it is 41 pages). The focus is largely on metrics for MT, but they also evaluate on image captioning. The paper is also very thorough and many of the questions I had when reading it are answered (like effect of optimal matching, running time, etc.). The latter (running time) being one of the downsides of the method if it was to be used for fine-tuning MT systems. 40 times slower than BLEU, but I think this increased cost would be worth it and could be engineered around.

Overall, I like the paper - it is simple and effective on its goal task of automatic evaluation for text generation. I think we are moving that way as a field and this paper proposes a useful method and is additionally a good study on the subject.

A question I have is why the method doesn't perform well in certain cases. For instance, in Table 2 and 3 - some of the evaluations with tr and fi fall well below relative performance for other language pairs. Does this have to do with the quality of the representations in multilingual BERT? What is YiSi-1 doing, for instance for model selection of en-fi and en-tr that makes it have so much better performance?

Edit: I also wonder if incorporating idf would be better if the values were computed by a larger corpus. I think it would make the most sense to compute these from the training data for the underlying BERT models. Since BERT itself is a function of this training data, it seems appropriate that these values would be as well (or perhaps at least a subset of this data).

Missing citations:
A citation to "Beyond BLEU:Training Neural Machine Translation with Semantic Similarity" from ACL 2019 should be incorporated into the related work. They use semantic similarity to fine-tune NMT systems with their own embedding-based (semantic similarity) metric and they found some nice properties from training in this way. Have you tried BERTScore on sentence similarity tasks? It's possible BERTScore could have strong performance and some readers may wonder this. There are evaluations on PAWS for paraphrase detection which I appreciated, but that is a little different.

A citation to "Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization" from EMNLP 2019 should also be incorporated. This paper is a big boon to BERT score showing that it is a very helpful metric for fine-tuning summarization systems. They don't even need a cross-entropy term since BERTScore captures fluency so well. I'd like to see it for MT as well, but perhaps that is the next paper.

Typos:
The word "language" is misspelled twice in Appendix E.
review_title: Official Blind Review #1

Meta-review: Thanks for an interesting discussion. The authors present a supposedly task-independent evaluation metric for generation tasks with references that relies on BERT or similar pretrained language models and a BERT-internal alignment. Reviewers are moderately positive. I encourage the authors to think about a) whether their approach scales to language pairs where wordpieces are less comparable; b) whether second order similarly, e.g., using RSA, would be better than alignment-based similarity; c) whether this metric works in the extremes, e.g., can it distinguish between bad output and super-bad output (where in both cases alignment may be impossible), and can it distinguish between good output and super-good output (where BERT scores may be too biased by BERT's training objective). 
Decision: Accept (Poster)