SlowGuess/NAACL_2022_OCR huggingface dataset:
Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chengguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, and Dragomir Radev. 2021. An exploratory study on long dialogue summarization: What works and what's next. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4426-4433, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Lulu Zhao, Weiran Xu, and Jun Guo. 2020. Improving abstractive dialogue summarization with graph structures and topic words. In Proceedings of the 28th International Conference on Computational Linguistics, pages 437-449, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China. Association for Computational Linguistics.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5927-5934, Online. Association for Computational Linguistics.
Yicheng Zou, Bolin Zhu, Xingwu Hu, Tao Gui, and Qi Zhang. 2021. Low-resource dialogue summarization with domain-agnostic multi-source pretraining. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 80–91, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
A Annotation Interface
Figure 1 and Figure 2 show the instructions for annotation and definition of each aspect. They were read by all annotators. Figure 3 shows a source dialogue and a summary to be evaluated.
B Correlation between different dimensions
Figure 4 shows the system-level correlation between coherence, consistency, fluency and relevance. Consistency is strongly correlated with relevance.
C Correlation between different metrics
Figure 5 shows the system-level correlation between different metrics.
D Reasons for discarding data from Amazon Mechanical Turk
Table 5 shows the result of model evaluation using annotations from Amazon Mechanical Turk. The performance of the models is indistinguishable, which is not consistent with our observation.
E The evaluation results for the models we reproduced
Table 6 shows the value of ROUGE-1, ROUGE-2 and ROUGE-L on the test set of SAMSum for the models we reproduced. The results is close to those in Gliwa et al. (2019) and Wu et al. (2021).
In this task you will evaluate the quality of summaries written for a dialogue from daily life.
To correctly solve this task, follow these steps:
1. Carefully read this dialogue, be aware of the information it contains.
2.Read the proposed summaries A-N (14 in total).
3. Rate each summary on a scale from 1 (worst) to 5 (best) by its relevance, consistency, fluency, coherence.
Figure 1: Instruction for annotators in data collection interface.
The rating measures how well the summary captures the key points of the dialogue.
Consider whether all and only the important aspects are contained in the summary.
The rating measures the whether the facts in the summary are consistent with the facts in the dialogue.
Consider whether the summary does reproduce all facts accurately and does not make up untrue information.
The rating measures the quality of individual sentences, are they well-written and grammatically correct.
Consider the quality of individual sentences.
The rating measures the quality of all sentences collectively, to the fit together and sound naturally.
Consider the quality of the summary as a whole.
Figure 2: Definition for annotators in data collection interface.
Marco: hi there! is this yours?
Marco:
Marco: somebody left it at my place yesterday
Sandra: oops yes its mine
Sandra: its Millas present, I bought it right before your party.
Sandra: can you bring it over?
sandra bought a present for milla right before marco's party and left it in his place . she asks marco to bring it over .
Relevance
- Consistency
Fluency
- Coherence
Figure 3: Annotation example in data collection interface.
Models	Coherence	Consistency	Fluency	Relevance
reference summary	3.308	3.300	3.396	3.380
LONGEST-3	3.220	3.230	3.286	3.306
LEAD-3	3.256	3.228	3.312	3.334
PGN	3.260	3.206	3.336	3.280
Tranformer	3.240	3.248	3.294	3.320
BART	3.286	3.298	3.410	3.358
PEGASUS	3.354	3.360	3.356	3.302
UniLM	3.288	3.342	3.390	3.364
CODS	3.346	3.328	3.384	3.396
ConvoSumm	3.368	3.334	3.420	3.426
MV-BART	3.232	3.260	3.366	3.344
PLM-BART	3.302	3.284	3.360	3.432
Ctrl-DiaSumm	3.232	3.300	3.360	3.348
S-BART	3.358	3.400	3.354	3.380
Table 5: Human ratings of summaries along four evaluation dimensions using data from Amazon Mechanical Turk. Scores are averaged over five annotators, broken down by the approximate classification in Section 3.3.
Models	ROUGE-1	ROUGE-2	ROUGE-L
LONGEST-3	30.60	9.61	27.96
LEAD-3	30.89	8.97	29.86
PGN	37.53	14.43	37.60
Tranformer	34.30	9.85	32.70
BART	52.59	28.43	50.16
PEGASUS	51.05	26.97	48.89
UniLM	49.43	24.26	47.21
Table 6: The results of automatic evaluation on the test set of SAMSum.
Figure 4: The correlation (Pearson's r) between different dimensions of human judgments on system level.
Figure 5: The correlation (Pearson's r) between different automatic evaluation metrics on system level.