SlowGuess/NAACL_2022_OCR huggingface dataset:
for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Elena Lloret, Laura Plaza, and Ahmet Aker. 2013. Analyzing the capabilities of crowdsourcing services for text summarization. Language resources and evaluation, 47(2):337-369.
Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267-300.
Jordan J Louviere and George G Woodworth. 1991. Best-worst scaling: A model for the largest difference judgments. Technical report, Working paper.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812-4829, Online. Association for Computational Linguistics.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2401–2410, Online. Association for Computational Linguistics.
Marta Sabou, Kalina Bontcheva, and Arno Scharl. 2012. Crowdsourcing research opportunities: lessons from natural language processing. In Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies, pages 1-8.
Sashank Santhanam and Samira Shaikh. 2019. Towards best experiment design for evaluating dialogue system output. In Proceedings of the 12th International Conference on Natural Language Generation, pages 88-94, Tokyo, Japan. Association for Computational Linguistics.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3246-3256, Hong Kong, China. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936. PMLR.
Julius Steen and Katja Markert. 2021. How to evaluate a summarizer: Study design and statistical analysis for manual linguistic quality evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1861-1875, Online. Association for Computational Linguistics.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339. PMLR.
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics, pages 5059-5069, Florence, Italy. Association for Computational Linguistics.
Besides the average model rank and average rating scores across BWS, LS-5, and LS-10 evaluations, we also provide standard deviations in Table 5.
To demonstrate our annotation template and facilitate future research, we show the interface for BWS annotations in Figures 3 and 4 and the interface for Likert annotations in Figures 5 and 6. We made use of the survey feature in Amazon Mechanical Turk (MTurk) to link to these Google Forms in Figure 7.
Models	CNN/DM	XSum
BWS	LS	LS-10	BWS	LS	LS10
PEGASUS	3.2302/1.150	3.8872/1.051	7.4103/2.160	3.2473/0.936	3.3501/1.334	6.2472/2.978
ProphetNet	3.1003/1.026	3.8604/0.992	7.2504/2.252	3.3602/1.102	3.2933/1.359	6.4272/3.038
BART	3.5931/1.113	4.0171/0.973	7.7271/2.090	3.5701/1.179	3.4332/1.338	6.9371/2.889
BERTSUM	3.0874/0.984	3.8633/1.037	7.4532/2.309	2.8274/0.993	2.7904/1.390	5.1634/3.202
Table 5: Average model rank, rating, and standard deviation across BWS, LS and  $LS_{10}$  evaluations.
Figure 3: Screenshot of the instruction page for BWS annotation.
Section 1/5
Summary 1
Which is the most factually consistent summary?
$\bigcirc$  Summary 1
$\bigcirc$  Summary 2
$\bigcirc$  Summary 3
$\bigcirc$  Summary 4
Which is the least factually consistent summary? *
Figure 4: Screenshot of the evaluation page for BWS annotation.
Page 3 of 7
Your task
* Required
Rate the summaries based on their factual consistency with the source. Factual consistency is rated on a five-point scale where 5 means perfect factual consistency and 1 means very poor factual consistency.
The factual consistency of a summary is determined by its agreement with facts in the source document. Factual consistency may not always relate to how good the summary is, though a factually inconsistent summary will certainly be a bad summary.
In some cases, you may find that the article and the summaries do not match, this may be due to the low quality of machine-generated summaries. Please indicate so at the end of the form with the section and the summary number.
How well do you understand the instructions? \*
Figure 5: Screenshot of the instruction page we used for Likert Scale annotation.
Page 2 of 7
Overall, how factually consistent do you find the summary with respect to the article? *
1. Very Poor; 2. Poor; 3. Barely Acceptable; 4. Good; 5. Very Good
Very Poor
Figure 6: Screenshot of the evaluation page for Likert Scale annotation.
Figure 7: This is how our task will look to Mechanical Turk Workers.