SlowGuess/NAACL_2021_OCR huggingface dataset:
Edward Sun*
edwsun@umich.edu
Dakuo Wang
Cambridge, USA
dakuo.wang@ibm.com
Yunfeng Zhang
Yorktown, USA
zhangyun@us.ibm.com
Nancy X.R. Wang
Almaden, USA
nancywang1991@gmail.com
Presentations are critical for communication in all areas of our lives, yet the creation of slide decks is often tedious and time-consuming. There has been limited research aiming to automate the document-to-slides generation process and all face a critical challenge: no publicly available dataset for training and benchmarking. In this work, we first contribute a new dataset, SciDuet, consisting of pairs of papers and their corresponding slides decks from recent years' NLP and ML conferences (e.g., ACL). Secondly, we present D2S, a novel system that tackles the document-to-slides task with a two-step approach: 1) Use slide titles to retrieve relevant and engaging text, figures, and tables; 2) Summarize the retrieved context into bullet points with long-form question answering. Our evaluation suggests that long-form QA outperforms state-of-the-art summarization baselines on both automated ROUGE metrics and qualitative human evaluation.
From business to education to research, presentations are everywhere as they are visually effective in summarizing and explaining bodies of work to the audience (Bartsch and Cobern, 2003; Wang, 2016; Piorkowski et al., 2021). However, it is tedious and time-consuming to manually create presentation slides (Franco et al., 2016).
Researchers have proposed various methods to automatically generate presentations from source documents. For example, Winters and Mathewson (2019) suggest heuristic rule-based mechanisms to extract document contents and use those as the generated-slide's content. PPSGen (Hu and Wan, 2014) leverages machine learning (ML) approaches
EXPERIMENTS - EFFECT OF THE AMOUNT OF AUXILIARY TASK DATA
Does our model heavily rely on the amount of auxiliary task data?
However, we do not observe substantial improvement when we further increase the sample rate.
Experiments effect of the amount of auxiliary task data
Figure 1: An example slide in SciDuet. TOP is author's original slide; BOTTOM is from our system D2S.
The performance goes up when we increase the sample rate from 0 to 0.2 for auxiliary task data.
Using only  $1\%$  auxiliary data, our model already obtains  $3.7\% -9.7\%$  absolute F-score gains
Figure: The effect of the amount of auxiliary task data on Dutch Name Tagging.
The performance goes up when we raise the sample rate from  $1\%$  to  $20\%$
However, we do not observe significant improvement when we further increase the sample rates.
- When trained with less than 50 main tasks training sentences, cross-lingual transfer consistently surpasses cross-task transfer.
- In the latter scheme, the linear layer and CRFs layer of the main model are not shared with other models.
to learn a sentence's importance in the document, and extract important sentences as slide's content.
These existing research works have yielded promising progress towards the goal of automated slide generation, but they also face two common limitations: 1) these works primarily rely on extractive-based mechanisms, thus the generated content is merely an aggregation of raw sentences from the document, whereas in real-world slides, the presenter frequently uses abstractive summarization; 2) these works assume the presentation slide's title has a one-to-one match to the document's subtitles or section headlines, whereas the presenter in reality often uses new slide titles and creates multiple slides under the same title (e.g.,
* Work done during internship at IBM Research.
Association for Computational Linguistics: Human Language Technologies, pages 1405-1418
the slides with a Cont. / Continue on it).
In our work, we aim to tackle both the limitations. To achieve this goal, we consider the document-to-slides generation task as a Query-Based Single-Document Text Summarization (QSS) task. Our approach leverages recent research developments from Open-Domain Long-form Question Answering (QA). Specifically, we propose an interactive two-step architecture: in the first step, we allow users to input a short text as the slide title and use a Dense Vector IR module to identify the most relevant sections/sentences as well as figures/tables from the corresponding paper. Then, in the second step, we use a QA model to generate the abstractive summary (answer) of the retrieved text based on the given slide title and use this as the final slide text content.
We design a keyword module to extract a hierarchical discourse structure from the paired paper. For a given title, we leverage leaf nodes from this tree structure in our IR module to rank paper snippets. We further extract related keywords from this structure and integrate them into the QA module. Experiments demonstrate that the keyword module helps our system to retrieve more relevant context and generate better slide content.
It is worth noting that our system can extract relevant figures and tables for a given title from the source document as well. Figure 1 (bottom) shows an example of a generated slide from our system.
In addition to our contribution of the novel model architecture, we also contribute a high-quality dataset (SciDuet), which contains 1,088 papers and 10,034 slides. We carefully build this dataset by leveraging a few toolkits for PDF parsing and image/table extraction. To the best of our knowledge, this is the first publicly available dataset for the document-to-slides generation task<sup>1</sup>. Our dataset together with the title-based document-to-slide generation task provide a practical testbed for the research field on query-based single-document summarization. We release the dataset procurement and preprocessing code as well as a portion of SciDuet<sup>2</sup> at https://github.com/IBM/document2slides.
The early works of automatically generating presentation slides date back to 20 years ago and rely on heuristic rule-based approaches to process information from web searches as slide contents for a user-entered topic (Al Masum et al., 2005). A recent example, Winters and Mathewson (2019) used predefined schemas, web sources, and rule-based heuristics to generate random decks based on a single topic. Among this group of works, different types of rules were used, but they all relied heavily on handcrafted features or heuristics (Shibata and Kurohashi, 2005; Prasad et al., 2009; Wang and Sumiya, 2013).
More recently, researchers started to leverage machine learning approaches to learn the importance of sentences and key phrases. These systems generally consist of a method to rank sentence importance: regression (Hu and Wan, 2014; Bhandare et al., 2016; Syamili and Abraham, 2017), random forest (Wang et al., 2017), and deep neural networks (Sefid et al., 2019). And they incorporate another method for sentence selection: integer linear programming (Hu and Wan, 2014; Sefid et al., 2019; Bhandare et al., 2016; Syamili and Abraham, 2017) and greedy methods (Wang et al., 2017). However, these methods all rely on extractive approaches, which extract raw sentences and phrases from the document as the generated slide content. An abstractive approach based on diverse titles that can summarize document content and generate new phrases and sentences is under-investigated.
Text Summarization To support abstractive document-to-slides generation, we refer to and are inspired by the Text Summarization literature. We consider the abstractive document-to-slide generation task as a query-based single-document text summarization (QSS) task. Although there has been increasing interest in constructing large-scale single-document text summarization corpora (CNN/DM (Hermann et al., 2015; Nallapati et al., 2016), Newsroom (Grusky et al., 2018), XSum (Narayan et al., 2018), TLDR (Cachola et al., 2020)) and developing various approaches to address this task (Pointer Generator (See et al., 2017), Bottom-Up (Gehrmann et al., 2018), BERTSum (Liu and Lapata, 2019)), QSS remains a relatively unexplored field. Most studies on query-based text summarization focus on the multi-document level
<sup>1</sup>Some previous works (SlideSeer (Kan, 2007), PPS-Gen (Hu and Wan, 2014) and (Wang et al., 2017)) described a dataset for training and testing, we could not obtain these datasets with our best ability to search and contact authors.  
<sup>2</sup>Due to copyright issues, we can only release a portion of our dataset. See Section 3 for more details. Other researchers can use our code to construct the full dataset from the original places or extend it with additional data.
(Dang, 2005; Baumel et al., 2016) and use extractive approaches (Feigenblat et al., 2017; Xu and Lapata, 2020). In the scientific literature domain, Erera et al. (2019) apply an unsupervised extractive approach to generate a summary for each section of a paper. In contrast to previous work, we construct a challenging QSS dataset for scientific paper-slide pairs and apply an abstractive approach to generate slide contents for a given slide title. In addition, Kryscinski et al. (2019) argues that future research on summarization should shift from "general-purpose summarization" to constrained settings. The new dataset and task we proposed provide the practical testbed to this end.
Open-Domain Long-Form Question Answering
Our work is motivated by the recent advancements in open-domain long-form question answering task, in which the answers are long and can span multiple sentences (ELI5 (Fan et al., 2019), NQ (Kwiatkowski et al., 2019)). Specifically, we consider the user-centered slide titles as questions and the paper document as the corpus. We use information retrieval (IR) to collect the most relevant text snippets from the paper for a given title before passing this to a QA module for sequence-to-sequence generation. We further improve the QA module by integrating title-specific key phrases to guide the model to generate slide content. In comparison to ELI5 and NQ, the questions in the slide generation task are shorter; and a significant proportion of the reference answers (slide contents) contain tables and figures directly from the paper, which then requires particular consideration.
3 SciDuet Dataset Construction
Data Sources The SciDuet (SClentic DocU-ment slidE maTch) dataset comprises of paperslide pairs scraped from online anthologies of International Conference on Machine Learning (ICML'19), Neural Information Processing Systems (NeurIPS'18&'19), and Association for Computational Linguistics (since ACL'79) conferences. We focus only on machine learning conferences as their papers have highly specialized vocabulary; we want to test the limits of language generation models on this challenging task. Nevertheless, these generic procurement methods (web-scraping) can be applied to other domains with structured archives.
Data Processing Text on papers was extracted through Grobid (GRO, 2008-2020). Figures and
#papers	#slides	ST-len	SC-len
train	952	8,123	3.6	55.1
dev	55	733	3.16	63.4
test	81	1,178	3.4	52.3
Table 1: Dataset statistic. 'ST-len' and 'SC-len' indicate the average token length for slide titles and slide contents, respectively.
% of novel n-grams
unigrams	bigrams	trigrams
SlideTitle	35.1	71.9	87.4
SlideContent	22.9	64.1	84.7
Table 2: The average proportion of novel n-grams for slide titles and slide contents in the training dataset.
captions were extracted through pdfigures2 (Clark and Divvala, 2016). Text on slides was extracted through IBM Watson Discovery package<sup>3</sup> and OCR by pytesseract.<sup>4</sup> Figures and tables that appear on slides and papers were linked through multiscale template matching by OpenCV. Further dataset cleaning was performed with standard string-based heuristics on sentence building, equation and floating caption removal, and duplicate line deletion.
Dataset Statistics and Analysis SciDuet has 952-55-81 paper-slide pairs in the Train-Dev-Test split. We publicly release SciDuet-ACL which is constructed from ACL Anthology. It contains the full Dev and Test sets, and a portion of the Train dataset. Note that although we cannot release the whole training dataset due to copyright issues, researchers can still use our released data procurement code to generate the training dataset from the online ICML/NeurIPS anthologies.
Table 1 shows the statistics of the dataset after excluding figures and tables from slide contents. In the training dataset,  $70\%$  of slide titles have fewer than five tokens, and  $59\%$  of slide contents have fewer than 50 tokens. $^{5}$
We also calculate the novel n-grams for slide titles and slide contents compared to the corresponding papers in the training dataset (Table 2). It seems that slide titles contain a higher proportion of novel n-grams compared to slide contents. Some exam
<sup>3</sup>https://www.ibm.com/cloud/watson-discovery
4 https://pypi.org/project/pytesseract
5Note that in Table 1, the Dev set has a slightly longer SC-len than the Train/Test sets. This is because there are two papers in the Dev set whose slides contain a lot of words. ST-Len in the Dev set is 56 after removing these two papers.
Figure 2: System architecture of our D2S framework.
plies of novel n-grams in slide titles are: recap, motivation, future directions, key question, main idea, and final remarks. Additionally, we found that only  $11\%$  of slide titles can match to the section and subsection headings in the corresponding papers.
4 D2S Framework
We consider document-to-slide generation as a closed-domain long-form question answering problem. Closed domain means the supporting context is limited to the paired paper. While traditional open-domain QA has specific questions, nearly  $40\%$  of our slide titles are generic (e.g., take home message, results). To generate meaningful slide contents (answers) for these titles (generic questions), we use title-like keywords to guide the system to retrieve and generate key bullet points for both generic titles and the specific keywords.
The system framework is illustrated in Figure 2. Below, we describe each module in detail.
4.1 Keyword Module
The inspiration for our Keyword Module is that paper often has a hierarchy structure and unspecified weak titles (e.g., Experiments or Results). We define weak titles as undescriptive generic titles nearly identical to section headers. The problem with these generic section headers is the length of their sections. Human presenters know to write content that spans the entire section. E.g., one may make brief comments on each subsection for a long Experiments section. For that, we use the keyword module to construct a parent-child tree of section
titles and subsection headings. We use this hierarchical discourse structure to aid our D2S system to improve information retrieval (Section 4.2) and slide content generation (Section 4.3).
4.2 Dense IR Module
Recent research has proposed various embedding-based retrieval approaches (Guu et al., 2020; Lee et al., 2019; Karpukhin et al., 2020) which outperform traditional IR methods like BM25. In our work, we integrate the leaf nodes of the parent-child trees from the keyword module into the reranking function of a dense vector IR system based on a distilled BERT miniature (Turc et al., 2019).
Without gold passage annotations, we train a dense vector IR model to minimize the cross-entropy loss of titles to their original content (taken from the original slides) because of their similarity to paper snippets. For a given title  $t$ , we randomly choose slide contents from other slides with different titles as the negative samples.
We precompute vector representations for all paper snippets (4 sentence passages) with the pretrained IR model. We then apply this model to compute a same-dimension dense vector representation for slide titles. Pairwise inner products are computed between the vectors of all snippets from a paper and the vector of a slide title. We use these inner products to measure the similarity between all title-snippet pairs, and we rank the paper passage candidates in terms of relevance to a given title with the help of Maximum Inner Product Search (Johnson et al., 2019). The top ten candidates are selected as input's context to the QA Module. We
further improve the IR re-ranking with extracted section titles and subsection headings (keywords) from the Keyword Module. We design a weighted ranking function with vector representations of titles, passage texts, and the leaf node keywords:
$$
\alpha (\mathsf {e m b} _ {\text {t i t l e}} \cdot \mathsf {e m b} _ {\text {t e x t}}) + (1 - \alpha) (\mathsf {e m b} _ {\text {t i t l e}} \cdot \mathsf {e m b} _ {\text {t e x t} _ {k w}})
$$
where  $emb_{title}$ ,  $emb_{text}$ , and  $emb_{text_{kw}}$  are the embedding vectors based on the pre-trained IR model for a given title, a text snippet, and the leaf node keyword from the keyword module which contains the text snippet, respectively. We find from the dev set that  $\alpha = 0.75$  is optimal. Experiments in Section 6 show that this ranking function can help our system become more header-aware and robust.
4.3 QA Module
The QA module in our D2S system combines slide title and the corresponding keywords as the query. It takes the concatenation of the top ten ranked text snippets from the IR module as the context.
We match a title to a set of keywords using the parent-child hierarchy from the keyword module. Note that this hierarchy is not limited to core sections  $(1,2,3,\ldots)$ , but can also be leveraged for all paper header matches x.x.x Specifically, if a title  $t$  matches with a header 2.1 (Levenshtein ratio  $\geq 0.9$ ), then we will include header 2.1 as well as all of its recursive children (e.g., 2.1.x, 2.1.x.x) as keywords for the QA module. It is worth noting that not every title has corresponding keywords.
Our QA model is a fine-tuned BART (Lewis et al., 2020). We encode the query and the context in the format of “{title[SEP1]keywords[SEP2]context}”. Keywords were embedded sequentially as a comma-separated list into the input following the slide title. We hypothesize that integrating keywords into the query can help our model pay attention towards relevant important context across all retrieved text fragments when generating slide content. This is indeed effective when the slide titles are aligned with broader sections, such as “Results”. In practice, embedding keywords helps the model in not just summarizing the top-ranked paragraphs, but also paying attention to additional paragraphs relevant to the broad topic.
We fine-tune our QA model using filtered training data. Filtering is done because the process of humans generating slides from a paper is highly creative and subjective to each author's unique style.
Some may include anecdotes or details outside the paired paper. These derivable lines, if not filtered, may hinder the QA module's performance on generating faithful sentences from the paired paper. Our experiments support this speculation.[6]
Training Data Filtering Due to the abstractive nature of slides, it is difficult to filter out slide content that is underviable from the paper content. No existing automated metrics can be used as a threshold to differentiate the derivable or underviable lines. To approach this, we performed manual gold standard annotations on 200 lines from slides to determine derivability. This led to the development of a Random Forest Classifier trained on the majority voting decision of annotators for 50 lines and tested on the remaining 150 lines. The classifier feature space is a combination of ROUGE-(1, 2, L) recall, precision, and F-scores. We apply this classifier to the original training set to filter out slide content that likely cannot be derived from the paired papers.
4.4 Figure Extraction Module
Slide decks are incomplete without good visual graphics to keep the audience attentive and engaged. Our D2S system adaptively selects connected figures and tables from the paper to build a holistic slide generation process. Our implementation is simple, yet effective. It reuses the dense vector IR module (Section 4.2) to compute vector similarities between the captions of figures/tables and the slide title (with the extended keywords if applicable). Figures and tables are then ranked and a final recommendation set is formed and presented to the user. This simulates an interactive figure recommendation system embedded in D2s.
Implementation Details All training was done on two 16GB P100 GPUs in parallel on PyTorch. Our code adapts the transformer models from HuggingFace (Wolf et al., 2020). All hyperparameters are fine-tuned on the dev set. A distilled uncased Bert miniature with 8-layers, 768 hidden units, and 12 attention heads was trained and used to perform IR. The BERT model computes all sentence embeddings in 128-dimensional vectors. Our QA model
Note that we always evaluate our model's performance on the unfiltered dataset (Section 8).
The derivability annotations together with the trained classifier can be accessed on our GitHub.
was fine-tuned over BART-Large-CNN, a BART model pre-trained on the CNN-Dailymail dataset. Pilot experiments showed that BART-Large-CNN outperforms BART-Large and other state-of-the-art pre-trained language generation models on the dev dataset. The BART model used the AdamW optimizer and a linear decreasing learning rate scheduler.
During testing, we apply our trained QA model with the lowest dev loss to the testing dataset. Note that we do not apply any content filtering on the testing data. The QA models generate the predicted slide content using beam search with beam 8 and no repeated trigrams. We use the dev dataset to tune the minimum and maximum token lengths for the output of the QA model.
Evaluation We evaluate our IR model using IDFrecall, which computes the proportion of words in the original slide text in the retrieved context weighted by their inverse document frequency. This metric gives more focuses to important words. For adaptive figure selection, we report the top-(1, 3, 5) precision. Finally, for slide text content generation, we use ROUGE as the automatic evaluation metric (Lin, 2004). We also carried out human evaluation to assess the D2S system's performance on slide text generation.
6 Evaluation on IR and Figure Selection
Results on Dense IR For a given slide title, the goal of IR is to identify relevant information from the paired paper for the downstream generation model. We compare our IR model (Dense-Mix IR) described in Section 4.2 to a few baselines. Classical IR (BM25) is based on sparse word matching which uses the BM25 similarity function. Dense-Text IR and Dense-Keyword IR are variants of our Dense-Mix IR model with different ranking functions ( $\alpha$  equals 1 for Dense-Text IR and 0 for Dense-Keyword IR).
All experiments are evaluated on the test set. The IDF-recall scores for each IR method are as follows: Classical IR (BM25) = 0.5112, Dense-Text IR = 0.5476, Dense-Keyword IR = 0.5175, and Dense-Mix IR = 0.5556.
The experiments indicate the dense IR model outperforming the classical IR approach and an  $\alpha = 0.75$ -weighted mix dense IR model outperforming other dense IR models that rank exclusively by text or keywords.
These results support the design decision of using embedding-based IR and re-ranking based on both text snippets and keywords. We attribute the success of the Dense-Mix IR model to increased section header awareness. Header-awareness leads to better retrieval in cases where the title corresponds well with section headers. The drawback of ranking solely on keywords is in the case when the dense IR module cannot differentiate between passages with the same header. This leads us to find the right balance  $(\alpha = 0.75)$  between Dense-Text IR and Dense-Keyword IR.
Results on Figure Selection We evaluate figure selection based on the set of the testing slides which contain figures/tables from the paired papers. The results of the adaptive figure selection are promising. It achieves 0.38, 0.60, and 0.77 on  $p@1$ ,  $p@3$ , and  $p@5$ , respectively. This suggests our system is holistic and capable of displaying figures and tables for slides that the original author chose.
7 Evaluation on Slide Text Generation
7.1 Baselines and BARTKeyword