SlowGuess/NAACL_2022_OCR huggingface dataset:
Mingqi Gao, Xiaojun Wan
{gaomingqi,wanxiaojun}@pku.edu.cn
Dialogue summarization is receiving increasing attention from researchers due to its extraordinary difficulty and unique application value. We observe that current dialogue summarization models have flaws that may not be well exposed by frequently used metrics such as ROUGE. In our paper, we re-evaluate 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency and relevance, as well as a unified human evaluation of various models in dialogue summarization for the first time. Some noteworthy trends which are different from the conventional summarization tasks are identified. We will release DialSummEval, a multi-faceted dataset of human judgments containing the outputs of 14 models on SAMSum.
Neural network based approaches and sizable datasets have led to significant progress in researches towards conventional summarization tasks such as news and scientific papers (Lin and Ng, 2019). Compared with conventional summarization tasks, dialogue summarization has received increasing attention from researchers due to its great difficulty and unique application value (Feng et al., 2021a). With the proposal of dialogue summary datasets such as SAMSum (Gliwa et al., 2019), DialogSum (Chen et al., 2021) and MediaSum (Zhu et al., 2021), a number of models for automatic generation of dialogue summaries have emerged (Feng et al., 2021b; Liu and Chen, 2021; Zou et al., 2021; Qi et al., 2021; Chen and Yang, 2020; Chen and Yang, 2021; Zhao et al., 2020; Liu et al., 2021).
There is no denying that these studies have made promising progress, but it remains a challenge to evaluate these advances comprehensively. Current studies generally use the SAMSum
dataset and adopt ROUGE (Lin, 2004), an n-gram-based automatic evaluation metric using reference summaries, as the overall evaluation criterion for summary quality, complemented by manual evaluation. Schluter (2017) and Graham (2015) illustrate the limitations of ROUGE in evaluating summarization tasks. Also the manual evaluation protocols vary from one research to another based on our observations.
We argue that the inadequate evaluation mechanism may have become a major obstacle to the progress of dialogue summarization researches. Many studies, such as Chen and Yang (2020) and Tang et al. (2021), have pointed out that the current dialogue summarization models still have many shortcomings, such as wrong references, incorrect reasoning and improper gender pronouns, and ROUGE may not reflect these problems effectively. For example, Gabriel et al. (2021) note that ROUGE-1 and ROUGE-L fail to accurately measure factual inconsistency across domains. Our case study in Table 1 also illustrates this point. However, it is impractical to perform frequent time-consuming and costly manual evaluation. The alternative is to introduce or propose more reliable automatic evaluation metrics to evaluate the models in a more comprehensive and fine-grained manner.
Although there are automatic evaluation metrics for measuring the quality of all aspects of summaries on conventional summarization tasks, especially for factual consistency (Huang et al., 2021), it is difficult to guarantee that they will still perform well on dialogue summarization. Recently proposed automatic metrics for evaluating generic natural language generation tasks such as BERTScore (Zhang* et al., 2020), BARTScore (Yuan et al., 2021) have also not been experimented on dialogue summarization. The high abstraction level, low extraction rate, and the requirement for complex reasoning power of the
$^{1}$ Code and data will be available at https://github.com/kite99520/DialSummEval
Human Language Technologies, pages 5693 - 5709
Dialogue	Reference Summary	Generated Summary	R-1	R-2	R-L
Kirsten: Youth group this Friday, don’t be late.
Alex: What time?	Kirsten reminds Alex that the youth group meets this friday at 7 pm and go bowling.	Kirsten is going bowling with her youth group this Friday at 7 pm.	0.69	0.44	0.61
Kirsten: 7 pm. We’re going bowling, so we’ll meet up and then all go together.
Alex: Cool. See you.
Kirsten: Bye
Ola: Hey running late	Ola should be free by 8. Kurt wants her to call him.	Ola will be late. She should be free by 8. Kurt will call her.	0.69	0.42	0.67
Ola: I should be free by 8
Kurt: Sure no prob, call me
Table 1: Case study of some outputs of BART on SAMSum. The ROUGE values of these outputs have substantially exceeded the state of the art on SAMSum. The summary in the first row fails in relevance, and the second has a factual error.
dialogue summarization task present new challenges to automatic evaluation metrics. There have been a number of manual evaluation datasets and analytical studies for conventional summarization tasks ((Dang and Owczarzak, 2008); Fabbri et al., 2021b; Bhandari et al., 2020), but very little work has been done on systematic analysis of dialogue summarization models and evaluation metrics. Our work will fill the gap in this area and includes the following contributions: 1) We identify evaluation problems in the field of dialogue summarization and point out the urgent need of automatic evaluation metrics that better adapt to dialogue summarization. 2) We collect and provide a sizable, multi-faceted dataset of manual evaluations for dialogue summarization, which contains the output of 14 models, and the dataset will be released. 3) We re-evaluate the performance of 18 types of automatic evaluation metrics on dialogue summarization. 4) We evaluate a variety of dialogue summarization models (extractive, abstractive, and recently based on pre-trained language models) in a unified manner.
Meta-Evaluation with Human Judgments Automatic evaluation Metrics such as ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020)) were compared with other metrics when proposed. However, they are basically not using the dialogue summarization dataset as an experimental corpus, and rarely provide new human judgments data. Bhandari et al. (2020) used pyramid (Nenkova
and Passonneau, 2004), a widely used human evaluation method on several conventional summarization datasets to obtain relevance scores for some of the system outputs and re-evaluated the metrics in 6 categories. Similarly, Fabbri et al. (2021b) used CNN/DailyMail dataset (Hermann et al., 2015) and the output of some models for human evaluation covering four facets of relevance, consistency, fluency, and coherence, and then re-evaluated the metrics in 14 categories. None of these involved dialogue summarization datasets. Pagnoni et al. (2021) made a careful categorization of factual errors and benchmarked factuality metrics using human annotations they collected on CNN/DailyMail and XSum dataset (Narayan et al., 2018). Notably, Gabriel et al. (2021) is one of the few current studies using the dialogue summarization dataset SAMSum (Gliwa et al., 2019) for meta-evaluation, but it focuses on factual consistency and selects a small number of metrics.
Analysis and Evaluation for Dialogue Summarization Models Tang et al. (2021) and Chen and Yang (2020) sampled the output of models on SAMSum and analyzes the error types when proposing a new model. Due to the different manual evaluation protocols and the small number of models included, it is difficult to comprehensively compare the strengths and weaknesses of different models. Khalifa et al. (2021) designed several tricks to address the special challenges in dialogue summarization and analyzed their effects, such as using name substitution to cope with the presence of multiple speakers in dialogues. Zhang
et al. (2021) focused on the problem of lengthy input and relevant information location in long dialogue summarization, and compared the performance of some models and strategies. No manual evaluation was involved in these studies.
In this section, we introduce the involved dataset, metrics and models.
SAMSum (Gliwa et al., 2019) is the first manually annotated, high-quality chat summarization dataset, containing over 16k dialogues. We use it in this study as it is most widely used and has greatly promoted the research in the field of dialogue summarization, and we are able to collect the outputs of various models on this dataset.
We selected a number of evaluation metrics that are frequently used on summarization or other natural language generation tasks. Some are for overall quality; others are specific to a particular aspect. Some require reference summaries or source documents; some only need the summary itself. Here is a brief categorization and description.
Metrics based on n-gram overlap include:
ROUGE (Lin, 2004) is the most widely used automatic evaluation metric in summarization. Researchers mainly adopt ROUGE-1, ROUGE-2 and ROUGE-L, which measure the unigram-overlap, bigram-overlap and longest common sequence between two texts respectively.
BLEU (Papineni et al., 2002) is the primary evaluation metric for machine translation. It calculates n-gram overlap between texts using precision scores and includes a brevity penalty. $^3$
METEOR (Banerjee and Lavie, 2005) computes an alignment by mapping unigrams in two texts, based on surface forms, stemmed forms, and meanings.
CHRF (Popovic, 2015) computes character based n-gram overlap between two texts.
Metrics based on pre-trained language models include:
BERTScore (Zhang* et al., 2020) measures the soft-overlap between two texts at token level using contextual embeddings from BERT.
MoverScore (Zhao et al., 2019) applies the semantic distance between two texts at n-gram level using n-gram embeddings pooled from BERT.
BARTScore (Yuan et al., 2021) treats evaluation as a nature language generation task and assumes that when the quality of generated text is better, the conditional language model has a higher probability of generating it from the source text or the reference, or is more likely to generate the reference from it. It can be flexibly applied to evaluation of text from different perspectives using BART.
BLANC (Vasilyev et al., 2020) is a referenceless metric. It hypothesizes that a good summary is beneficial for a pre-trained language model to conduct language understanding tasks on the source document. Specifically, it measures the performance boost of the masked language modeling for BERT utilizing the summary in two different ways. $^{8}$
PPL, namely perplexity, is often used to evaluate the quality of a language model or the fluency of an utterance. We adopt GPT-2 (Radford et al., 2019) as the language model for computing the perplexity for the whole summary.
Metrics based on word embeddings include:
SMS (Clark et al., 2019), namely Sentence Mover Similarity, extends Word Movers Distance (Kusner et al., 2015) to measure the distance between two texts which are represented as a bag of sentence embeddings.
Embedding average (Landauer and Dumais, 1997) is an embedding based metric computing the cosine similarity between the embeddings of two texts. A sentence-level embedding is represented by averaging the embeddings of the words composing the sentence.
Vector extrema (Forgues et al., 2014) is also an embedding based metric similar to Embedding av
$^{5}$ https://github.com/Tiiger/bert_score $^{6}$ https://github.com/AIPHES/emnlp19-moverscore
https://github.com/neulab/BARTScore  
https://github.com/PrimerAI/blanc  
https://huggingface.co/docs/  
transformers/perplexity
$^{2}$ https://github.com/Diego999/py-rouge
3Used code at https://github.com/Maluuba/nlg-eval, the same for Embedding average, Vector extrema, Greedy matching and METEOR, provided by (Sharma et al., 2017)
<sup>4</sup>https://github.com/m-popovic/chrF
erage. The metric computes a sentence-level embedding by taking the most extreme value of the embeddings of the words composing the sentence for each dimension of the embedding.
Greedy matching (Ru and Lintean, 2012) is another embedding based metric. The metric does not compute a sentence-level embedding. It directly compares the embeddings of words in the two sentences using a greedy matching algorithm to calculate similarity.
Metrics based on question-answering include:
FEQA (Durmus et al., 2020) employs a BERT-based question-answering model to answer questions using source document. Questions are generated by a fine-tuned BART model using generated summaries with masked named entities as inputs. The metric reports F1 scores against the gold answer, which are often regarded as a measure of factual consistency.[11]
SummaQA (Scialom et al., 2019) is also a QA-based metric. Unlike FEQA, it generates questions from source documents instead of summaries to be evaluated and then uses summaries to answer them. The F1 overlap score and QA-model confidence are reported. $^{12}$
QuestEval (Scialom et al., 2021) is another a QA-based metric. This metric can be considered as a combination of FEQA and SummaQA. It takes into account the scores obtained from both styles. For comparison purposes, We use the reference-less mode. $^{13}$
Metrics based on entailment classification include:
FactCC (Kryscinski et al., 2020) is a metric based on entailment classification. We follow the way Pagnoni et al. (2021) used it. Each sentence of the summary is fed into the classifier together with the document to determine whether the facts are consistent, and the proportion of consistent sentences is used to indicate how consistent the summary is.
DAE (Goyal and Durrett, 2020; Goyal and Durrett, 2021) is an entailment classification metric based on dependencies. We use it in a similar way
to FactCC. When a sentence cannot be parsed by the metric, we default it factually inconsistent.[15]
3.3 Summarization Models
We select some representative models and get the outputs of them on the test set of SAMSum. We choose LEAD-3 and LONGEST-3 as representatives of the simple extractive approaches. PGN (See et al., 2017) and Transformer (Vaswani et al., 2017) are selected as representatives of the earlier neural summarization models. For generic pretrained generative models, we use BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020) and UniLM (Dong et al., 2019). We retrain these models above to obtain the outputs and the automatic evaluation results are close to Gliwa et al. (2019) and Wu et al. (2021) in default settings. For models specifically designed for dialogue summarization, we choose CODS (Wu et al., 2021), ConvoSumm (Fabbri et al., 2021a), MV-BART (Chen and Yang, 2020), PLM-BART (Feng et al., 2021c), Ctrl-DiaSumm (Liu and Chen, 2021), S-BART (Chen and Yang, 2021) and the outputs are all provided by their authors. We also regard the reference summary as a kind of model output.
4 Data Annotation
4.1 Annotation Setup
Since human evaluation is expensive and time-consuming, we decide to randomly sample 100 dialogues from the test set of SAMSum and evaluate the summaries generated by all models on these dialogues. To comprehensively evaluate each metric and model, we perform human evaluation in four aspects, as in Kryscinski et al. (2019):
Coherence measures the quality of all sentences in the summary as a whole. It focuses on whether the summary is coherent and natural.
Consistency measures how well the summary aligns with the dialogue in facts. It focuses on whether the summary contains factual errors.
Fluency measures the quality of individual sentences in the summary compared to Coherence. It focuses on whether the sentences are well-written and grammatically correct.
Relevance measures how well the summary captures the key points of the dialogue. It focuses on whether all and only the important aspects are contained in the summary.
15https://github.com/tagoyal/ factuality-datasets
11https://github.com/esdurmus/feqa 12https://github.com/ThomasScialom/ summa-qa 13https://github.com/ThomasScialom/ QuestEval 14https://github.com/salesforce/factCC
To ensure the quality of the annotation, we tried to annotate some of the data ourselves at the beginning to judge the difficulty of the task and the approximate time spent.
We initially tried to annotate the data using crowdsourcing platforms. We published the annotation task on Amazon Mechanical Turk  $^{16}$ . The interface contained instructions and definitions of the four aspects. A dialogue and a corresponding summary were included in the interface, and the summaries of different models on the same dialogue were presented to the annotators in a sequence to facilitate comparison. For each dimension/aspect, annotators were asked to rate the summary on a Likert scale from 1 to 5. Each summary was evaluated by 5 different annotators, and For each dimension we would receive a total of  $100 \times 14 \times 5 = 7000$  human annotations. The annotation was done quickly in one day, but the quality was not satisfactory. We calculated the average score of each model in each aspect based on these annotation data and found that the scores of the models are close in each dimension, which is not in the accordance with the reality. For example, in terms of consistency, the reference summary and the extractive approaches should have had a definite advantage, but this failed to be reflected from the data. The result is shown in Table 5. For reliability reasons, we do not use these annotations for our analysis.
Then, we decided to recruit annotators from the school forum who are required to be capable of reading daily conversations and articles in English fluently. We recruited three annotators, using a similar annotation interface and approach as in the crowd-sourcing platforms. These annotators were college students and they are fluent in English. The differences with the crowd-sourcing platform annotation are as follows: 1) For a student who wanted to participate in the annotation, we would ask him to annotate all models on the first 10 conversations  $(10 \times 14 = 140$  annotations), and let her/him continue the annotation only when these annotation results were checked by us to confirm that the annotator had understood the task correctly and could finish the annotation responsibly. Otherwise, we paid the annotator directly for this part and terminated his annotation task. 2)
We required each annotator to annotate all data  $(100 \times 14 = 1400$  annotations) to ensure the consistency within the annotator. 3) During the annotation process, we kept in touch with the annotators via email or instant messaging app to answer their questions at any time.
It took around 10 days to finish the annotation. We received  $100 \times 14 \times 3 = 4200$  annotations for each perspective. For each aspect of each summary, if two scores were the same and the other was different from them, we considered the different one as noise. For each dimension, we removed the noise separately and calculated the Krippendorff's alpha coefficient (Krippendorff, 2011). We found the inter-annotator interval metric to be within an acceptable range - from 0.5621 to 0.7564, as detailed in Table 2. The raw annotated data will be released and we use the cleaned data for analysis. At last, we use the average of the cleaned data to represent the human evaluation score of an summary on a dimension.
5 Metric Evaluation
In this section, we will introduce several definitions in meta-evaluation and re-evaluate the metrics mentioned in Section 3.2.
As mentioned by Bhandari et al. (2020), there are two common ways to measure the correlation of automatic evaluation metrics to manual evaluation: system-level and summary-level.
Assuming there are  $N$  dialogues, the  $i$ -th dialogue is represented as  $d_{i}$ . For a dialogue  $d_{i}$ , there are  $J$  summaries generated by  $J$  models, and we denote each of them as  $s_{ij}, j = 1 \dots J$ . There are  $K$  evaluation metrics (or human evaluation) in total, and  $m_{k}$  refers to an automatic evaluation metric or human evaluation of a certain dimension.  $m_{k}(s_{ij})$  means the score of  $k$ -th metric towards a pair of dialogue and summary ( $d_{i}, s_{ij}$ ). We use  $R(m_{i}, m_{j})$  to denote the correlation coefficient between two metrics  $m_{i}$  and  $m_{j}$ .
System-level correlation is defined as follows. The corresponding p-value which indicates statistical significance can be obtained:
$^{16}$ https://www.mturk.com
Coherence	Consistency	Fluency	Relevance
cleaned	3161	3360	3050	3439
total	4200	4200	4200	4200
Krippendorff's alpha	0.7564	0.6709	0.6782	0.5621
Table 2: The inter-annotator agreement for each dimension.
$$
R _ {s y s} (m _ {p}, m _ {q}) = R (
$$
$$
\left[ \frac {1}{N} \sum_ {i = 1} ^ {N} m _ {p} \left(s _ {i 1}\right), \dots , \frac {1}{N} \sum_ {i = 1} ^ {N} m _ {p} \left(s _ {i J}\right) \right],
$$
$$
[ \frac {1}{N} \sum_ {i = 1} ^ {N} m _ {q} (s _ {i 1}), \dots , \frac {1}{N} \sum_ {i = 1} ^ {N} m _ {q} (s _ {i J}) ])
$$
Summary-level correlation is defined as follows, and the p-value cannot be derived here because the Summary-level correlation is an average value:
$$
R _ {s u m} (m _ {p}, m _ {q}) = \frac {1}{N} \sum_ {i = 1} ^ {N} R (
$$
$$
[ m _ {p} (s _ {i 1}), \dots , m _ {p} (s _ {i J}) ],
$$
$$
\left[ m _ {q} \left(s _ {i 1}\right), \dots , m _ {q} \left(s _ {i J}\right) \right])
$$
Comparing the performance of various metrics reveals some trends in Table 3. In each dimension, metrics which are strongly correlated with human judgments exist, but few metrics show significant strengths in all four dimensions. Of all the metrics, QuestEval has the most comprehensive capabilities at the system level. Generally metrics that perform better on coherence and fluency perform worse on consistency and relevance, and vice versa. This can be attributed to the definition of the dimensions, i.e. there is some correlation between the four dimensions themselves, which is shown in Figure 4. In all dimensions, automatic evaluation metrics based on pre-trained language models generally outperform metrics based on n-gram overlap and context-independent word embedding. Among them, the recently proposed BARTScore and the increasingly popular QA-based metrics perform the best. This suggests that both directions have the potential to be explored in terms of
evaluation for dialogue summarization. Across dimensions, almost all metrics correlate better with human judgments at the system level than at the summary level, and both showed good agreement with each other. This indicates that the summary-level correlations are also worth referring to when enough data are not available for system-level analysis. In addition, metrics such as BLEU and CHRF, which are frequently used in other natural language generation tasks (e.g., machine translation, dialogue, etc.), do not show advantages on dialogue summarization.
The characteristics presented by the automatic evaluation metrics on the dialogue summarization differ from those of the conventional summarization tasks. For ROUGE, we find that increasing the size of n in ROUGE-n is not better in almost all dimensions, which is different from the findings of Rankel et al. (2013) and Fabbri et al. (2021b). The ability of ROUGE to reflect content selection, i.e., relevance, as we usually believe, is also questionable. Compared to the results of Fabbri et al. (2021b), metrics based on n-gram overlap such as ROUGE and CHRF perform worse on dialogue summarization, while some metrics that use source documents such as BLANC perform better. We need to focus on the limitations of ROUGE and the role of the source dialogues in evaluating dialogue summaries.
We have also observed some interesting phenomena. Entailment classification metrics such as FactCC and DAE outperform many metrics in terms of consistency, but not as well as BARTScore and QA-based metrics. This may be due to the large gap between the corpus used in training and dialogues, and the need to slice the summaries by sentence when using them. FEQA, which is designed for factual consistency, however, performs best in coherence and fluency, and rather poorly in consistency and relevance. Comparing its performance with QuestEval and SummaQA, generating questions from the original dialogue may be more reliable in measuring consistency, which corroborates with the points of Gabriel et al.