SlowGuess/NAACL_2022_OCR huggingface dataset:
Coherence	Consistency	Fluency	Relevance
Metrics	sys	sum	sys	sum	sys	sum	sys	sum
ROUGE-1	0.59*	0.30	0.42	0.33	0.58*	0.27	0.40	0.30
ROUGE-2	0.47	0.26	0.41	0.32	0.43	0.22	0.41	0.30
ROUGE-3	0.39	0.22	0.39	0.30	0.33	0.17	0.40	0.30
ROUGE-4	0.33	0.20	0.37	0.27	0.27	0.14	0.38	0.28
ROUGE-L	0.57*	0.32	0.39	0.30	0.54*	0.27	0.37	0.27
BERTScore-p	0.57*	0.37	0.11	0.10	0.50	0.31	0.08	0.06
BERTScore-r	0.43	0.21	0.45	0.38	0.42	0.20	0.46	0.39
BERTScore-f1	0.53	0.31	0.28	0.24	0.48	0.27	0.27	0.22
MoverScore	0.50	0.28	0.39	0.32	0.46	0.25	0.38	0.31
SMS	0.33	0.18	0.38	0.28	0.27	0.14	0.40	0.29
BARTScore-s-h +	0.09	0.08	0.62*	0.44	0.24	0.15	0.60*	0.42
BARTScore-h -	0.08	0.05	-0.09	-0.09	0.16	0.13	-0.18	-0.12
BARTScore-h-r	0.50	0.21	0.55*	0.46	0.51	0.21	0.56*	0.46
BARTScore-r-h	0.67**	0.42	0.31	0.23	0.67**	0.40	0.26	0.17
BLANC-help +	-0.32	-0.21	0.54	0.45	-0.13	-0.08	0.60*	0.50
BLANC-tune +	-0.37	-0.23	0.50	0.38	-0.18	-0.10	0.56*	0.43
FEQA +	0.82**	0.27	0.32	0.16	0.84**	0.26	0.25	0.10
QuestEval +	0.50	0.15	0.85**	0.39	0.75**	0.20	0.83**	0.37
SummaQA-conf +	-0.08	-0.03	0.64*	0.39	0.03	-0.01	0.67**	0.39
SummaQA-fscore +	-0.26	-0.11	0.58*	0.26	-0.06	-0.06	0.62*	0.29
PPL -	-0.13	-0.01	-0.49	-0.30	-0.34	-0.15	-0.43	-0.30
CHRF	0.42	0.20	0.46	0.38	0.41	0.20	0.47	0.39
BLEU-1	0.35	0.15	0.34	0.29	0.30	0.13	0.36	0.30
BLEU-2	0.31	0.16	0.35	0.29	0.25	0.12	0.37	0.30
BLEU-3	0.28	0.15	0.33	0.27	0.21	0.11	0.36	0.28
BLEU-4	0.25	0.14	0.33	0.25	0.17	0.09	0.36	0.28
METEOR	0.37	0.19	0.42	0.35	0.33	0.17	0.43	0.35
Embedding average	0.43	0.17	0.17	0.20	0.52	0.22	0.15	0.19
Vector extrema	0.47	0.22	0.35	0.28	0.43	0.21	0.35	0.26
Greedy matching	0.43	0.21	0.35	0.31	0.43	0.21	0.36	0.30
FactCC +	-0.29	-0.09	0.46	0.19	-0.23	-0.09	0.49	0.19
DAE +	-0.24	-0.07	0.50	0.29	-0.15	-0.02	0.54*	0.29
Table 3: The correlation (Pearson's r) of annotations computed on system level and summary level along four quality dimensions between automatic metrics and human judgments. For evaluation, all metrics require at least the summaries to be evaluated as input. Metrics with + indicate that the source dialogues are used, metrics with - means no other input are required, others need to use the reference summaries. The five most-correlated metrics in each column are bolded (For system level,  $**$ =significant for  $p \leq 0.01$ ,  $*$ =significant for  $p \leq 0.05$ ). We add suffixes to distinguish the different variants of metrics. For BARTScore, h, r and s are abbreviations of hypotheses, references and source dialogues respectively. BARTScore-s-h measure the probability to generate hypotheses using source dialogues as inputs, while BARTScore-h measures the probability to generate hypotheses without other inputs, and so on. For BLANC, BLANC-tune refers to the way of fine-tuning on a generated summary and then conducting nature language understanding tasks on source dialogues, while BLANC-help refers to the way of inferring with a generated summary concatenated together. For SummaQA, SummaQA-fscore measures the average overlap between predictions and ground truth answers, and SummaQA-conf corresponds to the confidence of the predictions.
Models	Coherence	Consistency	Fluency	Relevance	R-1	R-2	R-L
reference summary	4.500	4.370	4.560	4.210	1.000	1.000	1.000
LONGEST-3	3.230	4.393	4.100	4.363	0.304	0.099	0.267
LEAD-3	4.370	4.093	4.200	3.843	0.309	0.092	0.296
PGN	3.568	2.103	3.657	2.293	0.356	0.126	0.357
Tranformer	3.403	1.573	3.673	1.650	0.329	0.098	0.319
BART	4.480	3.667	4.667	3.500	0.533	0.299	0.520
PEGASUS	4.590	3.730	4.640	3.417	0.508	0.254	0.476
UniLM	4.303	3.320	4.523	3.290	0.489	0.232	0.470
CODS	4.268	3.637	4.567	3.397	0.523	0.278	0.509
ConvoSumm	4.507	3.743	4.643	3.437	0.532	0.268	0.498
MV-BART	4.320	3.937	4.660	3.747	0.539	0.290	0.513
PLM-BART	4.360	3.717	4.680	3.500	0.533	0.284	0.507
Ctrl-DiaSumm	4.320	3.893	4.650	3.670	0.564	0.312	0.549
S-BART	4.227	3.307	4.520	3.337	0.497	0.244	0.472
Table 4: Human ratings of summaries along four evaluation dimensions using cleaned annotations from campus recruitment. Scores are averaged over annotators for a summary, and scores are averaged over all summaries for a model. The table is broken down by the approximate classification in Section 3.3. For comparison, ROUGE values calculated using our sampling data are also shown. Please note that this may differ from the results in the original literature. The two highest-rated models in each column are in bold.
(2021). It is surprising that metrics based on the language model such as PPL, BARTScore-h performs poorly in measuring both coherence and fluency. The exact reasons for this need further investigation.
6 Model Evaluation
In each dimension, we evaluate each model mentioned in Section 3.3 using the average of the human evaluation scores of all summaries. Analyzing Table 4, we conclude the following.
The reference summaries in SAMSum are not perfect, and the annotators felt that they also contained some factual inconsistencies compared to the source dialogues, as well as important elements of the dialogues that were not all captured by them. However, comparing the human evaluation scores of the reference summaries in CNNDM (Fabbri et al., 2021b), the quality is already superior.
Extractive models produce summaries that differ in style from abstractive models, and many conversations contain ungrammatical utterances, which can affect the reading experience and impair their fluency and coherence. In particular, LONGEST-3, which extracts some potentially discontinuous sentences from dialogues, has low coherence. However, since they do not modify the content, they still perform well in terms of con
sistency. Since the average length of dialogues in SAMSum is small, extracting a few sentences from it can generally include important contents, so the relevance is also high. The evaluation of the extractive models raises a question: what kind of summaries do readers actually want?
The early neural summarization models represented by PGN and Transformer perform relatively poorly in all dimensions compared to the reference summaries, especially consistency and relevance. This is to be expected because of the high difficulty of dialogue summarization and the small size of SAMSum dataset.
An important finding is that the generic pretrained language models represented by BART, PEGASUS and UniLM, and various recently proposed models specifically designed on the dialogue summarization task do not have significant differences in each dimension. They are already comparable, and in some cases better, in terms of coherence and fluency compared to the reference summaries. They have improved dramatically compared to earlier neural summarization models with respect to consistency and relevance, but there is still some room for enhancement. On the one hand, this finding affirms the capability of these models; On the other hand, it urges us to reflect on how much these recently proposed complex models or fancy techniques are an improvement over the generic pre-trained language mod
els.
We point out the problems with the evaluation in the dialogue summarization and introduce DialSummEval, a multi-faceted dataset containing the output of various models and the corresponding human judgments. Based on this dataset, we provide a comprehensive re-evaluation and analysis of the performance of widely used automatic evaluation metrics and each model. There are three important findings: 1) Few metrics are excellent in all dimensions, and the recently proposed BARTScore and QA-based metrics are comparatively outstanding and worth exploring. 2) The automatic evaluation metrics and their variants present some trends that differ from conventional summarization. 3) A variety of models specifically designed for dialogue summarization perform comparably to reference summaries in terms of coherence and fluency, but still have shortcomings in consistency and relevance. We hope that researchers in the field recognize the importance of evaluation in current research, choose some other metrics in addition to ROUGE when evaluating models, propose automatic evaluation metrics that can be better adapted to the field of dialogue summarization based on our work.
Whether recruiting annotators through Amazon Mechanical Turk or campus, we paid them 15 dollars per hour, more than the local average minimum wage. We removed all content in the dataset that might contain personal information about the annotators.
We thank all authors for providing the summaries generated by their models. We thank Baizhou Huang for his help in the process of retraining some models. This work was supported by National Science Foundation of China (No. 62161160339), National Key RD Program of China (No.2018YFB1005100), State Key Laboratory of Media Convergence Production Technology and Systems and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers
for their helpful comments. Xiaojun Wan is the corresponding author.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Reevaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online. Association for Computational Linguistics.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118, Online. Association for Computational Linguistics.
Jiaao Chen and Diyi Yang. 2021. Structure-aware abstractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1380-1391, Online. Association for Computational Linguistics.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062-5074, Online. Association for Computational Linguistics.
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748-2760, Florence, Italy. Association for Computational Linguistics.
Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the tac 2008 update summarization task. In TAC.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021a. ConvoSumm: Conversation summarization benchmark and improved abstractive summarization with argument mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6866-6880, Online. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021b. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021a. A survey on dialogue summarization: Recent advances and new frontiers. Computing Research Repository, arXiv:2107.03175. Version 1.
Xiachong Feng, Xiaocheng Feng, Bing Qin, and Xinwei Geng. 2021b. Dialogue discourse-aware graph model and data augmentation for meeting summarization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3808-3814. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021c. Language model as an annotator: Exploring DialogGPT for dialogue summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1479-1491, Online. Association for Computational Linguistics.
Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevéque, and Réal Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2.
Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021. GO FIGURE: A meta evaluation of factuality in summarization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 478-487, Online. Association for Computational Linguistics.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive
summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3592-3603, Online. Association for Computational Linguistics.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449-1462, Online. Association for Computational Linguistics.
Yvette Graham. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128-137, Lisbon, Portugal. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28:1693-1701.
Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. The factual inconsistency problem in abstractive text summarization: A survey. Computing Research Repository, arXiv:2104.14839. Version 2.
Muhammad Khalifa, Miguel Ballesteros, and Kathleen McKeown. 2021. A bag of tricks for dialogue summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8014-8022, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 957-966, Lille, France. PMLR.
Thomas K Landauer and Susan T. Dumais. 1997. A solution to Plato's problem: the latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review, pages 211-240.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Hui Lin and Vincent Ng. 2019. Abstractive summarization: A survey of the state of the art. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):9815-9822.
Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021. Topic-aware contrastive learning for abstractive dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1229-1243, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 92-106, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Massachusetts, USA. Association for Computational Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812-4829, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
MengNan Qi, Hao Liu, YuZhuo Fu, and Ting Liu. 2021. Improving abstractive dialogue summarization with hierarchical pretraining and topic segment. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1121-1130, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Peter A. Rankel, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2013. A decade of automatic content evaluation of news summaries: Reassessing the state of the art. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 131-136, Sofia, Bulgaria. Association for Computational Linguistics.
Vasile Rus and Mihai Lintean. 2012. An optimal assessment of natural language student input using word-to-word similarity metrics. In International Conference on Intelligent Tutoring Systems, pages 675-676. Springer.
Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41-45, Valencia, Spain. Association for Computational Linguistics.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3246-3256, Hong Kong, China. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, and Dragomir Radev. 2021. Confit: Toward faithful dialogue summarization with linguistically-informed contrastive fine-tuning. Computing Research Repository, arXiv:2112.08713. Version 1.
Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11-20, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus Stenetorp, and Caiming Xiong. 2021. Controllable abstractive dialogue summarization with sketch supervision. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5108-5122, Online. Association for Computational Linguistics.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Computing Research Repository, arXiv:2106.11520. Version 2.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.