Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework
Year: 2023

Abstract:
Factuality is important to dialogue summarization. Factual error correction (FEC) of modelgenerated summaries is one way to improve factuality. Current FEC evaluation that relies on factuality metrics is not reliable and detailed enough. To address this problem, we are the first to manually annotate a FEC dataset for dialogue summarization containing 4000 items and propose FERRANTI, a fine-grained evaluation framework based on reference correction that automatically evaluates the performance of FEC models on different error categories. Using this evaluation framework, we conduct sufficient experiments with FEC approaches under a variety of settings and find the best training modes and significant differences in the performance of the existing approaches on different factual error categories. 1

heading: Future Work
text: For future work, it is feasible to apply FERRANTI to FEC for other summarization tasks. Limitations
Due to limited resources, the size of our annotated dataset is not large, with only 4000 items. In addition, we use an annotation paradigm where direct writing is the main focus with error labeling as a supplement. This is good for the coherence of the corrected summary and gives larger freedom to the annotator. In this case, it may be better to increase the number of reference corrections per sample. The datasets we select, SAMSum and DialogSum, are both short daily chat summarization datasets