SlowGuess/NAACL_2021_OCR huggingface dataset:
Below we describe the technical details of the two baselines as well as our QA module (BARTKeyword) for slide text generation.
BertSummExt From (Liu and Lapata, 2019), the model is fine-tuned to the retrieved context on our unfiltered training dataset. For a given title and the retrieved context based on our IR model, the model extracts important sentences from the context as the slide text content. Note that performance was lowered with filtering, which differs from other models. We suspect that the extractive model depends on output text lengths. Filtering reduces the ground truth token length, which in turn, makes the generated output also shorter, leading to a marginally higher precision at greater cost in recall. Hyperparameters are reused from (Liu and Lapata, 2019) and training continues from the best pre-trained weights of the CNN/Daily-Mail task. This maintains consistency with the Bart models, which were also pre-trained on CNN/DM.
BARTSumm A BART summarization model fine-tuned on the filtered dataset. We use a batch size of 4 with an initial learning rate of  $5e - 5$ . We set the maximum input token length at 1024, which is approximately the same length as the retrieved context (10 paper snippets  $\approx$  40 sentences, each
sentence  $\approx 25$  tokens). Min and max output token lengths were found to be 50 and 128.
Our Method (BARTKeyword) This is our proposed slide generation model as described in Section 4.3. We fine-tune our QA model on the filtered dataset with a batch size of 4 and an initial learning rate of  $5e - 5$ . The maximum input token length was also set to 1024. Dev set tuned min and max token lengths were found to be 64 and 128.
7.2.1 Automated Evaluation
We use ROUGE scores to evaluate the generated content with regard to the ground-truth slide content. Overall, our Dense-Mix IR approach provides better context for the downstream summarization models. In general, our BARTKeyword model is superior to the abstractive and extractive summarization models in all ROUGE metrics (1/2/L) based on different IR approaches as shown in Table 3. Additionally, the abstractive summarization model performs better than the extractive model. Altogether, this shows the importance of adopting an abstractive approach as well as incorporating the slide title and keywords as additional context for better slide generation from scientific papers.
Human-Generated Slides (Non-Author) As presentation generation is a highly subjective task, we wanted to estimate the expected ROUGE score that a non-author (but subject domain expert) human may be able to obtain by generating slides from the paper. In total, three authors of this paper each randomly selected and annotated one paper (either from dev or test), and another common paper (Paper ID: 960), thus in total four papers have non-author human-generated slides. $^{8}$  The procedure we followed was: read the paper thoroughly, and then for each slide's original title, generate high-quality slide content using the content from the paper. The high quality of our non-author experts generated slides can be demonstrated through the high scores given for the human-generated slides in the human evaluation (Section 7.2.2).
Table 4 shows the results of ROUGE F-score for non-author generated slides compared to our D2S system. It is interesting to see that our model's performance is similar to or sometimes better than the non-author generated ones. The task of generating slides from a given paper is indeed difficult
even for subject domain experts, which is quite a common task in "research paper reading groups". It is easy for humans to miss important phrases and nuances, which may have resulted in the lower score compared to the model.
In general, the low human annotator ROUGE F-score shown in Table 4 reflects the difficulty and subjectivity of the task. This result also provides a reasonable performance ceiling for our dataset.
Four Models As suggested in the ACL'20 Best Paper (Ribeiro et al., 2020), automatic evaluation metrics alone cannot accurately estimate the performance of an NLP model. In addition to the automated evaluation, we also conducted a human evaluation to ask raters to evaluate the slides generated by BARTKeyword (our model), by baseline models (both BARTSumm and BertSummExt) based on Dense-Mix IR, and by the non-author human experts (Human).
Participant The human evaluation task involves reading and rating slides from the ACL Anthology. We noted that some technical background was required, so we recruited machine learning researchers and students ( $N = 23$ ) with snowball sampling. These participants come from several IT companies and universities. Among them: 10 have more than 3 years of ML experience; 7 have more than 1 year; 13 actively work on NLP projects; and 7 know the basic concepts of NLP.
Dataset In the human evaluation, we use 81 papers from the test set. We filter out papers with fewer than 8 slides, as each rater will do 8 rounds in an experiment, leaving 71 papers in the set.
Task We follow prior works' practices of recruiting human raters to evaluate model-generated contents (Wang et al., 2021). For each rater, we randomly select two papers, one from the former four papers, and another one from the test set. For each paper, we again randomly select four slides, thus each participant complete eight rounds of evaluation (2 papers  $\times$  4 slides). In each round, a participant rates one slide's various versions from different approaches with reference to the original author's slide and paper. The participants rate along three dimensions with a 6-point Likert scale (1 strongly disagree to 6 strongly agree):
- Readability: The generated slide content is coherent, concise, and grammatically correct;
Manually generated slides available on our GitHub.
Summarization Model	ROUGE-1	ROUGE-2	ROUGE-L
P	R	F	P	R	F	P	R	F
Classical IR (BM25)
BertSummExt	14.26	24.07	15.89	2.59	4.46	2.86	12.89	21.70	14.31
BARTSumm	15.75	23.40	16.92	2.94	4.12	3.11	14.18	20.99	15.55
BARTKeyword (ours)	17.15	27.98	19.06	4.08	6.52	4.52	16.29	24.88	18.12
Dense-Mix IR (ours)
BertSummExt	15.47	25.74	17.16	3.14	5.24	3.47	13.97	23.29	15.48
BARTSumm	16.62	26.10	18.15	3.35	5.16	3.63	15.00	23.28	16.73
BARTKeyword (ours)	18.30	30.31	20.47	4.73	7.79	5.26	16.86	27.21	19.08
Table 3: ROUGE scores of our BARTKeyword QA model compared to other summarization baselines based on different IR approaches.
Generator	R-1	R-2	R-L
Humans	26.41	8.66	24.68
D2S	27.75	8.30	24.69
Table 4: ROUGE F-scores for non-author generated slides in comparison to our  $D2S$  system.
- Informativeness: The generated slide provides sufficient and necessary information that corresponds to the given slide title, regardless of its similarity to the original slide;
- Consistency: The generated slide content is similar to the original author's reference slide.
Result Ratings on the same model's slides are aggregated into an average, resulting in three scores for each of the four models (three systems plus Human). ANOVA tests are used for each dimension (Greenhouse-Geisser correction applied when needed) to compare the models' performances, and a post hoc pairwise comparisons with Tukey's honest significance difference (HSD) test (Field, 2009).
Results show that for the Readability dimension (Figure 3), the BertSummExt model performs significantly worse than the other three models  $(F(1.77, 39.04) = 6.80, p = .004)$ , and that between the three models there is no significant difference. This result suggests that even though the extraction-based methods use grammatically correct sentences from the original paper, the human raters do not think the content is coherent or concise; however, it also indicates that summarization-based models can achieve fairly high readability.
The most Informative slides were generated by humans ( $F(1.59, 35.09) = 13.10$ ,  $p < .001$ ). But BARTkeyword (our model) came in second and outperformed BertSummExt significantly ( $t(66) = 3.171$ ,  $p = .012$ ) and BARTsumm insignificantly
Figure 3: Average rating given by participants to each method across three dimensions. Error bars represent  $95\%$  confidence intervals.
$(t(66) = 2.171, p = .142)$ . Our model is also the only ML model rated above 3.5 (the midpoint of the 6-point Likert scale), meaning on average, participants agree that the model is informative.
Regarding the Consistency between the generated slide content and the author's original slide, there is a significant difference in ratings across methods,  $F(1.68, 37.03) = 30.30$ ,  $p < .001$ . Human-generated slides outperformed the ML models again in this metric, but BARTKeyword also significantly outperformed the other two:  $t(66) = 4.453$ ,  $p < .001$  vs BertSummExt, and  $t(66) = 2.858$ ,  $p = 0.028$  vs BARTSumm. This indicates that our model provides a SOTA performance in the consistency dimension, but there is space to improve to reach the human level.
8 System Analysis
In this section, we carry out additional experiments to better understand the effectiveness of different components in our system.
R-1	R-2	R-L
BARTKeyword
Oracle-IR	36.32	16.99	36.59
Dense-Mix IR	20.47	5.26	19.08
Dense-Mix IR
BARTKeyword	20.47	5.26	19.08
BaseQA	20.19	5.04	18.81
Dense-Mix IR + BARTKeyword
Filter	20.47	5.26	19.08
Unfilter	20.02	4.94	18.64
Table 5: ROUGE F-scores of system variations.
IR-Oracle To estimate an upper-bound of the ROUGE score, we design an IR model to locate the best context possible for retrieval. For each line in the ground-truth slide content, we retrieve the most related sentence from the entire paper scored by a weighted ROUGE score. This oracle model sees information that would not be available in a regular slide generation system, which only has the title as input. Similar to what was shown in the human annotation experiment (Table 4), the F-score in all ROUGE metrics remain below 40 (Table 5, row Oracle-IR), demonstrating the subjectiveness of the task and providing context for the level of performance achieved by the D2S System.
Effect of keywords in Summarization Section 6 shows that our keywords aware Dense-Mix IR model achieves the best IDF-recall score on the test dataset. Here we test the effect of keywords in the QA module. Table 5 shows that removing keywords from BARTKeyword (BaseQA) leads to performance degradation. It seems that the extracted keywords for a given title can help our model to locate relevant context from all retrieved text snippets and generate better content.
Effect of Dataset Filtering in Summarization We also test the effect of filtering the training dataset in the QA module. Table 5 shows that training BARTKeyword on the filtered training dataset (described in Section 4.3) helps improve performance in the unfiltered test set. This is likely due to the reduction of noisy text that cannot be generated from the document, allowing the model to learn to synthesize information from the text without trying to hallucinate new information.
9 Error Analysis
To gain additional insights into our model's performance, we carried out a qualitative error analysis to check the common errors in our best system (Dense-Mix IR + BARTKeyword). We sampled 20 slides that received lower rating scores (rating score  $< 3$  in at least one dimension) in our human evaluation experiment (Section 7.2.2). One author of this paper carefully checked each generated slide content and compared it to the original paper/slide.
In general, we found that most errors are due to off-topic content. For instance, given a slide title "Future Work", our model might generate sentences that summarize the major contributions of the corresponding paper but do not discuss next steps. We also observed that occasionally our model hallucinates content which is not supported by the corresponding paper. Normally, this happens after the model selects an example sentence from the paper and the sentence's content is very different from its surrounding context. For instance, a paper uses an example sentence "Which cinema screen Star Wars tonight?" to illustrate a new approach to capture intents/slots in conversations. Then for the slide title "Reminder Q&A Data", our model generates "Which cinema screen Star Wars tonight? Which movie theater plays Star Wars at 8 p.m. on December 18?" Here, the second sentence is a hallucination error.
We use the novel n-grams to measure the "abstractiveness" of the generated slide contents. On the testing dataset, we found that the original slide contents contain a much higher proportion of novel n-grams compared to the automatically generated ones (e.g.,  $24.2\%$  vs.  $3.1\%$  for novel unigrams, and  $66.5\%$  vs.  $14.2\%$  for novel bigrams). This indicates that the generated slide contents from our model are still mostly "extractive".
This project aims to automatically generate presentation slides from paper documents. The problem is framed as a query-based single-document summarization task. Inspired by recent work on open-domain long-form QA, we design a keyword-aware framework (D2S) to tackle this challenge. Both automated and human evaluations suggest that our system outperforms a few strong baselines and can be served as a benchmark for the document-to-slide challenge. We release the dataset (SciDuet) and code in hopes it can foster future work.
The authors appreciate the valuable feedback from the anonymous reviewers. We also thank our friends and colleagues who participated in our human evaluation study.
2008-2020. Grobid. https://github.com/ kermitt2/grobid.
Shaikh Mostafa Al Masum, Mitsuru Ishizuka, and Md Tawhidul Islam. 2005. 'auto-presentation': a multi-agent system for building automatic multimodal presentation of a topic from world wide web information. In IEEE/WIC/ACM International Conference on Intelligent Agent Technology, pages 246-249. IEEE.
Robert A Bartsch and Kristi M Cobern. 2003. Effectiveness of powerpoint presentations in lectures. Computers & education, 41(1):77-86.
Tal Baumel, Raphael Cohen, and Michael Elhadad. 2016. Topic concentration in query focused summarization datasets. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, page 2573-2579. AAAI Press.
Anuja A Bhandare, Chetan J Awati, and Sonam S Kharade. 2016. Automatic era: Presentation slides from academic paper. In 2016 International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT), pages 809-814. IEEE.
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld. 2020. TLDR: Extreme summarization of scientific documents. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4766-4777, Online. Association for Computational Linguistics.
Christopher Clark and Santosh Divvala. 2016. PDF figures 2.0: Mining figures from research papers. In 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL), pages 143-152. IEEE.
H. Dang. 2005. Overview of duc 2005.
Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, Ora Peled Nakash, Odellia Boni, Haggai Roitman, Doron Cohen, Bar Weiner, Yosi Mass, Or Rivlin, Guy Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Francesca Bonin, and David Konopnicki. 2019. A summarization system for scientific documents. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 211-216, Hong Kong, China. Association for Computational Linguistics.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki. 2017. Unsupervised query-focused multi-document summarization using the cross entropy method. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, page 961-964, New York, NY, USA. Association for Computing Machinery.
Andy Field. 2009. Discovering statistics using SPSS: (and sex and drugs and rock 'n'roll). Sage.
P Regnath Franco, K Thirumoorthy, and K Muneeswaran. 2016. Automatic creation of well-organized slides from documents. In 2016 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), pages 1173-1177. IEEE.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. ArXiv, abs/2002.08909.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28, pages 1693-1701. Curran Associates, Inc.
Yue Hu and Xiaojun Wan. 2014. Ppsgen: Learning-based presentation slides generation for academic papers. IEEE transactions on knowledge and data engineering, 27(4):1085-1097.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Min-Yen Kan. 2007. Slideseer: A digital library of aligned document and presentation pairs. In Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, pages 81-90.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
David Piorkowski, Soya Park, April Yi Wang, Dakuo Wang, Michael Muller, and Felix Portnoy. 2021. How ai developers overcome communication challenges in a multidisciplinary team: A case study. arXiv preprint arXiv:2101.06098.
K Gokul Prasad, Harish Mathivanan, TV Greetha, and M Jayaprakasam. 2009. Document summarization and information extraction for generation of presentation slides. In 2009 International Conference on Advances in Recent Technologies in Communication and Computing, pages 126-128. IEEE.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Athar Sefid, Jian Wu, Prasenjit Mitra, and C Lee Giles. 2019. Automatic slide generation for scientific papers. In SciKnow@ K-CAP, pages 11-16.
Tomohide Shibata and Sadao Kurohashi. 2005. Automatic slide generation based on discourse structure analysis. In International Conference on Natural Language Processing, pages 754-766. Springer.
S Syamili and Anish Abraham. 2017. Presentation slides generation from scientific papers using support vector regression. In 2017 International Conference on Inventive Communication and Computational Technologies (ICICCT), pages 286-291. IEEE.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv: Computation and Language.
Dakuo Wang. 2016. How people write together now: Exploring and supporting today's computer-supported collaborative writing. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, pages 175-179.
Liuping Wang, Dakuo Wang, Feng Tian, Zhenhui Peng, Xiangmin Fan, Zhan Zhang, Shuai Ma, Mo Yu, Xiaojuan Ma, and Hongan Wang. 2021. Cass: Towards building a social-support chatbot for online health community. CSCW'21.
Sida Wang, Xiaojun Wan, and Shikang Du. 2017. Phrase-based presentation slides generation for academic papers. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 196-202.
Yuanyuan Wang and Kazutoshi Sumiya. 2013. A method for generating presentation slides based on expression styles using document structure. International Journal of Knowledge and Web Intelligence, 4(1):93-112.
Thomas Winters and Kory W Mathewson. 2019. Automatically generating engaging presentation slide decks. In International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar), pages 127-141. Springer.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
A.1 Model Training and Parameters