SlowGuess/NAACL_2022_OCR huggingface dataset:
Sajad Sotudeh and Nazli Goharian
IR Lab, Georgetown University, Washington DC 20057, USA
{sajad, nazli}@ir.cs.georgetown.edu
Many scientific papers such as those in arXiv and PubMed data collections have abstracts with varying lengths of 50-1000 words and average length of approximately 200 words, where longer abstracts typically convey more information about the source paper. Up to recently, scientific summarization research has typically focused on generating short, abstract-like summaries following the existing datasets used for scientific summarization. In domains where the source text is relatively long-form, such as in scientific documents, such summary is not able to go beyond the general and coarse overview and provide salient information from the source document. The recent interest to tackle this problem motivated curation of scientific datasets, arXiv-Long and PubMed-Long, containing human-written summaries of 400-600 words, hence, providing a venue for research in generating long/extended summaries. Extended summaries facilitate a faster read while providing details beyond coarse information. In this paper, we propose TSTR, an extractive summarizer that utilizes the introductory information of documents as pointers to their salient information. The evaluations on two existing large-scale extended summarization datasets indicate statistically significant improvement in terms of ROUGE and average ROUGE (F1) scores (except in one case) as compared to strong baselines and state-of-the-art. Comprehensive human evaluations favor our generated extended summaries in terms of cohesion and completeness.
Over the past few years, summarization task has witnessed a huge deal of progress in extractive (Nallapati et al., 2017; Liu and Lapata, 2019; Yuan et al., 2020; Cui et al., 2020; Jia et al., 2020; Feng et al., 2018) and abstractive (See et al., 2017; Cohen et al., 2018; Gehrmann et al., 2018; Zhang et al., 2019; Tian et al., 2019; Zou et al., 2020)
[Introductory] Neural machine translation (@xcite), directly applying a single neural network to transform the source sentence into the target sentence, has now reached impressive performance (@xcite [...] Motivated by recent success in unsupervised cross-lingual embeddings (@xcite), the models proposed for unsupervised NMT often assume that a pair of sentences from two different languages can be mapped to a same latent representation in a shared-latent space (@xcite) [...]. Although the shared encoder is vital for mapping sentences from different languages into the shared-latent space, it is weak in keeping the uniqueness and internal characteristics of each language, such as the style, terminology and sentence structure. [...] For each language, the encoder and its corresponding decoder perform an AE, where the encoder generates the latent representations from the perturbed input sentences and the decoder reconstructs the sentences from the latent representations. Experimental results show that the proposed approach consistently achieves great success.
[Non-introductory] [...] To further enforce the shared-latent space, we train a discriminative neural network, referred to as the local discriminator, to classify between the encoding of source sentences and the encoding of target sentences. [...] the shared encoder is weak in keeping the unique characteristic of each language. This confirms our intuition that the shared layers are vital to map the source and target latent representations to a shared-latent space. [...] This shows that the proposed model only trained with monolingual data effectively learns to use the context information and the internal structure of each language [...] The models proposed recently for unsupervised NMT use a single encoder to map sentences from different languages to a shared-latent space. [...] The experimental results reveal that our approach achieves significant improvement and verify our conjecture that the shared encoder is really a bottleneck for improving the unsupervised NMT.
Figure 1: A truncated human-written extended summary. Top box: introductory information, bottom box: non-introductory information. Colored spans are pointers from introductory sentences to associated non-introductory detailed sentences.
settings. Many scientific papers such as those in arXiv and PubMed (Cohan et al., 2018) possess abstracts of varying length, ranging from 50 to 1000 words and average length of approximately 200 words. While scientific paper summarization has been an active research area, most works (Cohan et al., 2018; Xiao and Carenini, 2019; Cui and Hu, 2021; Rohde et al., 2021) in this domain have focused on generating typical short and abstract-like summaries (Chandrasekaran et al., 2020). Short summaries might be adequate when the source text is of short-form such as those in news domain; however, to summarize longer documents such as scientific papers, an extended summary including 400-600 terms on average, such as those found in extended summarization datasets of arXiv-Long and PubMed-Long, is more appealing as it conveys more detailed information.
Human Language Technologies, pages 325 - 335
Extended summary generation has been of research interest very recently. Chandrasekaran et al. (2020) motivated the necessity of generating extended summaries through LongSumm shared task  $^{1}$ . Long documents such as scientific papers are usually framed in a specific structure. They start by presenting general introductory information  $^{2}$ . This introductory information is then followed by supplemental information (i.e., non-introductory) that explain the initial introductory information in more detail. Similarly, as shown in Figure 1, this pattern holds in a human-written extended summary of a long document, where the preceding sentences (top box inside Figure 1) are introductory sentences and succeeding sentences (bottom box inside Figure 1) are explanations of the introductory sentences. In this study, we aim to guide the summarization model to utilize the aforementioned rationale in human-written summaries. We consider introductory sentences as those that appear in the first section of paper with headings such as Introduction, Overview, Motivations, and so forth. As such, all other parts of paper and their sentences are considered as non-introductory (i.e., supplementary). We use these definitions in the reminder of this paper.
Herein, we approach the problem of extended summary generation by incorporating the most important introductory information into the summarization model. We hypothesize that incorporating such information into the summarization model guides the model to pick salient detailed non-introductory information to augment the final extended summary. The importance of the role of introduction in the scientific papers was earlier presented in (Teufel and Moens, 2002; Armagan, 2013; Jirge, 2017) where they showed such information provides clues (i.e. pointers) to the objectives and experiments of studies. Similarly, Boni et al. (2020) conducted a study to show the importance of introduction part of scientific papers as its relevance to the paper's abstract. To validate our hypothesis, we test the proposed approach on two publicly available large-scale extended summarization datasets, namely arXiv-Long and PubMedLong. Our experimental results improve over the strong baselines and state-of-the-art models. In short, the contributions of this work are as follows:
- A novel multi-tasking approach that incorporates the salient introductory information into the extractive summarizer to guide the model in generating a 600-term (roughly) extended summary of a long document, containing the key detailed information of a scientific paper.
- Intrinsic evaluation that demonstrates statistically significant improvements over strong extractive and abstractive summarization baselines and state-of-the-art models.
- An extensive human evaluation which reveals the advantage of the proposed model in terms of cohesion and completeness.
Summarizing scientific documents has gained a huge deal of attention from researchers, although it has been studied for decades. Neural efforts in scientific text have used specific characteristics of papers such as discourse structure (Cohan et al., 2018; Xiao and Carenini, 2019) and citation information (Qazvinian and Radev, 2008; Cohen and Goharian, 2015, 2018) to aid summarization model. While prior work has mostly covered the generation of shorter-form summaries (approx. 200 terms), generating extended summaries of roughly 600 terms for long-form source documents such as scientific papers has been motivated very recently (Chandrasekaran et al., 2020).
The proposed models for the extended summary generation task include jointly learning to predict sentence importance and sentence section to extract top sentences (Sotudeh et al., 2020); utilizing section-contribution computations to pick sentences from important section for forming the final summary (Ghosh Roy et al., 2020); identifying salient sections for generating abstractive summaries (Gidiotis et al., 2020); ensembling of extraction and abstraction models to form final summary (Ying et al., 2021); an extractive model with TextRank algorithm equipped with BM25 as similarity function (Kaushik et al., 2021); and incorporating sentences embeddings into graph-based extractive summarizer in an unsupervised manner (Ramirez-Orta and Milios, 2021). Unlike these works, we do not exploit any sectional nor citation information in this work. To the best of our knowledge, we are the first at proposing the novel method of utilizing introductory information of the scientific paper to guide the model to learn to generate
'https://ornlcda.github.io/SDProc/sharedtasks.html
2We will exchangeably use (non-)introductory information and (non-)introductory sentences in the rest of this paper.
summary from the salient and related information.
3 Background: Contextualized language models for summarization
Contextualized language models such as BERT (Devlin et al., 2019), and ROBERTA (Liu et al., 2019) have achieved state-of-the-art performance on a variety of downstream NLP tasks including text summarization. Liu and Lapata (2019) were the first to fine-tune a contextualized language model (i.e., BERT) for the summarization task. They proposed BERTSUM—a fine-tuning scheme for text summarization—that outputs the sentence representations of the source document (we use the term source and source document interchangeably, referring to the entire document). The BERTSUMEXT model, which is built based on BERTSUM, was proposed for the extractive summarization task. It utilizes the representations produced by BERTSUM, passes them through Transformers encoder (Vaswani et al., 2017), and finally uses a linear layer with Sigmoid function to compute copying probabilities for each input sentence. Formally, let  $l_{1}, l_{2}, \ldots, l_{n}$  be the binary tags over the source sentences  $\mathbf{x} = \{sent_{1}, sent_{2}, \ldots, sent_{n}\}$  of a long document, in which  $n$  is the number of sentences in the paper. The BERTSUMEXT network runs over the source documents as follows (Eq. 1),
$$
h _ {b} = \operatorname {B e r t S u m} (\mathbf {x})
$$
$$
h = \operatorname {E n c o d e r} _ {\mathbf {t}} \left(h _ {b}\right) \tag {1}
$$
$$
p = \sigma \left(W _ {o} h + b _ {o}\right)
$$
where  $\mathbf{h}_{\mathbf{b}}$  and  $\mathbf{h}$  are the representations of source sentences encoded by BERTSUM and Trasformers encoder, respectively.  $W_{o}$  and  $b_{o}$  are trainable parameters, and  $p$  is the probability distribution over the source sentences, signifying extraction copy likelihood. The goal of this network is to train a network that can identify the positive sets of sentences as the summary. To prevent the network from selecting redundant sentences, BERTSUM uses TriGram Blocking (Liu and Lapata, 2019) for sentence selection in inference time. We refer the reader to the main paper for more details.
4 TSTR: Intro-guided Summarization
In this section, we describe our methodology to tackle the extended summary generation task. Our approach exploits the introductory information<sup>3</sup>.
Figure 2: Our model uses introductory sentences as pointers to the source sentences. It then forms the final extended summary by extracting salient sentences from the source. Highlights in red show the salient parts.
of the paper as pointers to salient sentences within it, as shown in Figure 2. It is ultimately expected that the extractive summarizer is guided to pick salient sentences across the entire paper.
The detailed illustration of our model is shown in Figure 3. To aid the extractive summarization model (i.e., right-hand box in Figure 3) which takes in source sentences of a scientific paper, we utilize an additional BERTSUM encoder called Introductory encoder (left-hand box in Fig. 3) that receives  $\mathbf{x}_{\mathrm{intro}} = \{sent_1, sent_2, \dots, sent_m\}$ , with  $m$  being the number of sentences in introductory section. The aim of adding second encoder in this framework is to identify the clues in the introductory section which point to the salient supplementary sentences<sup>4</sup>. The BERTSUM network computes the extraction probabilities for introductory sentences as follow (same way as in Eq. 1),
$$
\tilde {h} _ {b} = \operatorname {B e r t S u m} \left(\mathbf {x} _ {\text {i n t r o}}\right)
$$
$$
\tilde {h} = \mathbf {E n c o d e r} _ {\mathbf {t}} \left(\tilde {h} _ {b}\right) \tag {2}
$$
$$
\tilde {p} = \sigma (W _ {j} \tilde {h} + b _ {j})
$$
in which  $\tilde{h}_b$ , and  $\tilde{h}$  are the introductory sentence representations by BERTSUM, Transformers encoder, respectively.  $\tilde{p}$  is the introductory sentence extraction probabilities.  $W_{j}$  and  $b_{j}$  are trainable matrices.
After identifying salient introductory sentences, the representations associated with them are retrieved using a pooling function and further used to guide the first task (i.e., right-hand side in Figure 3) as follows,
$$
\tilde {h} _ {t o p} = \text {S e l e c t} (\tilde {h}, \tilde {p}, k) \tag {3}
$$
$$
\hat {h} = \mathbf {M L P _ {1}} (\tilde {h} _ {t o p})
$$
3Introductory information is defined in Section 1
4Supplementary sentences are defined in Section 1.
Figure 3: Detailed illustration of our summarization framework. Task-1  $(\mathbf{t}_1)$ : source sentence extraction (right-hand gray box). Task-2  $(\mathbf{t}_2)$ : introductory sentence extraction (left-hand gray box). As shown, the identified salient introductory sentences at training stages are incorporated into the representations of source sentences by the Select( $\cdot$ ) function (orange box) with  $k = 3$ . Plus sign shows the concatenation layer. The feed-forward neural network is made of one linear layer.
where Select  $(\cdot)$  is a function that takes in all introductory sentence representations (i.e.,  $\tilde{h}$ ), and introductory sentence probabilities  $\tilde{p}$ . It then outputs the representations associated with top  $k$  introductory sentences, sorted by  $\tilde{p}$ . To extract top introductory sentences, we first sort  $\tilde{h}$  vectors based on their computed probabilities  $\tilde{p}$  and then we pick up top  $k$  hidden vectors (i.e.,  $\tilde{h}_{top}$ ) that has the highest probability.  $\mathbf{MLP_1}$  is a multi-layer perceptron that takes in concatenated vector of top introductory sentences and projects it into a new vector called  $\hat{h}$ .
At the final stage, we concatenate the transformed introductory top sentence representations (i.e.,  $\hat{h}$ ) with each source sentence representations from Eq. 1 (i.e.,  $h_i$  where  $i$  shows the  $i$ th paper sentence) and process them to produce a resulting vector  $r$  which is intro-aware source sentence hidden representations. After processing the resulting vector through a linear output layer (with  $W_z$  and  $b_z$  as trainable parameters), we obtain final intro-aware sentence extraction probabilities (i.e.,  $p$ ) as follows,
$$
r = \mathbf {M L P} _ {\mathbf {2}} (h _ {i}; \hat {h})
$$
$$
p = \sigma \left(W _ {z} r + b _ {z}\right) \tag {4}
$$
in which  $\mathbf{MLP}_2$  is a multi-layer perceptron, influencing the knowledge from introductory sentence extraction task (i.e.,  $\mathbf{t}_2$ ) into the source sentence extraction task (i.e.,  $\mathbf{t}_1$ ). We train both tasks through our end-to-end system jointly as follows,
$$
\ell_ {\text {t o t a l}} = (\alpha) \ell_ {\mathbf {t} _ {1}} + (1 - \alpha) \ell_ {\mathbf {t} _ {2}} \tag {5}
$$
where  $\ell_{\mathbf{t}_1}$ , and  $\ell_{\mathbf{t}_2}$  are the losses computed for introductory sentence extraction and source sentence extraction tasks,  $\alpha$  is the regularizing parameter that balances the learning process between two tasks, and  $\ell_{\mathrm{total}}$  is the total computed loss that is optimized during the training.
In this section, we explain the datasets, baselines, and preprocessing and training parameters.
We use two publicly available scientific extended summarization datasets (Sotudeh et al., 2021).
- arXiv-Long: A set of arXiv scientific papers containing papers from various scientific domains such as physics, mathematics, computer science, quantitative biology. arXiv-Long is intended for extended summarization task and was filtered from a larger dataset i.e., arXiv (Cohan et al., 2018) for the summaries of more than 350 tokens. The ground-truth summaries (i.e., abstract) are long, with the average length of 574 tokens. It contains 7816 (train), 1381 (validation), and 1952 (test) papers.
- PubMed-Long: A set of biomedical scientific papers from PubMed with average summary length of 403 tokens. This dataset contains 79893 (train), 4406 (validation), and 4402 (test) scientific papers.
- LongSumm: The recently proposed LongSumm dataset for a shared task (Chandrasekaran et al., 2020) contains 2236 abstractive and extractive summaries for training and 22 papers for the official test set. We report a comparison with BERTSUMEXTMULTI using this data in Table 2. However, as the official test set is blind, our experimental results in Table 1 do not use this dataset.
We compare our model with two strong non-neural systems, and four state-of-the-art neural summarizers. We use all of these baselines for the purpose of extended summary generation whose documents hold different characteristics in length, writing style, and discourse structure as compared to documents in the other domains of summarization.
- LSA (Steinberger and Ježek, 2004): an extractive vector-based model that utilizes Singular Value Decomposition (SVD) to find the semantically important sentences.
- LEXRANK (Erkan and Radev, 2004): a widely adopted extractive summarization baseline that utilizes a graph-based approach based on eigenvector centrality to identify the most salient sentences.
- BERTSUMEXT (Liu and Lapata, 2019): a contextualized summarizer fine-tuned for summarization task, which encodes input sentence representations, and then processes them through a multi-layer Transformers encoder to obtain
document-level sentence representation. Finally, a linear output layer with Sigmoid activation function outputs a probability distribution over each input sentence, denoting the extent to which they are probable to be extracted.
- BERTSUMEXT-INTRO (Liu and Lapata, 2019): a BERTSUMEXT model that only runs on the introductory sentences as the input, and extracts the salient introductory sentences as the summary.
- BERTSUMEXTMULTI (Sotudeh et al., 2021): an extension of the BERTSUMEXT model that incorporates an additional linear layer with Sigmoid classifier to output a probability distribution over a fixed number of pre-defined sections that an input sentence might belong to. The additional network is expected to predict a single section for an input sentence and is trained jointly with BERTSUMEXT module (i.e., sentence extractor).
- BART (Lewis et al., 2020): a state-of-the-art abstractive summarization model that makes use of pretrained encoder and decoder. BART can be thought of as an extension of BERTSUM in which merely encoder is pre-trained, but decoder is trained from scratch. While our model is an extractive one, at the same time, we find it of value to measure the abstractive model performance in the extended summary generation task.
5.3 Preprocessing, parameters, labeling, and implementation details
We used the open implementation of BERT-SUMEXT with default parameters  $^{5}$ . To implement the non-neural baseline models, we utilized Sumy python package  $^{6}$ . Longformer model (Beltagy et al., 2020) is utilized as our contextualized language model for running all the models due to its efficacy at processing long documents. For our model, the cross-entropy loss function is set for two tasks (i.e.,  $\mathbf{t}_1$ : source sentence extraction and  $\mathbf{t}_2$ : introductory sentences extraction in Figure 3) and the model is optimized through multi-tasking approach as discussed in Section 3. The model with the highest ROUGE-2 on validation set is selected for inference. The validation is performed every 2k training steps.  $\alpha$  (in Eq. 5) is set to be 0.5 (empirically determined). Our model includes 474M
5https://github.com/nlpyang/PreSumm
$^{6}$ https://github.com/miso-belica/sumy
trainable parameters, trained on dual GeForce GTX 1080Ti GPUs for approximately a week. We use  $k = 5$  for arXiv-Long,  $k = 8$  for PubMed-Long datasets (Eq. 3). We make our model implementation as well as sample summaries publicly available to expedite ongoing research in this direction<sup>7</sup>.
A two-stage labeling approach was employed to identify ground-truth introductory and non-introductory sentences. In the first stage, we used a greedy labeling approach (Liu and Lapata, 2019) to label sentences within the first section of a given paper (i.e., labeling introductory sentences) with respect to their ROUGE overlap<sup>8</sup> with the ground-truth summary (i.e., abstract). In the second stage, the same greedy approach was exploited over the rest of sentences (i.e., non-introductory)<sup>9</sup> with regard to their ROUGE overlap with the identified introductory sentences in the first stage. Our choice of ROUGE-2 and ROUGE-L is based on the fact that these express higher similarity with human judgments (Cohan and Goharian, 2016). We continued the second stage until a fixed length of the summary was reached. Specifically, the fixed length of positive labels is set to be 15 for arXiv-Long, and 20 for PubMed-Long datasets as these achieved the highest oracle ROUGE scores in our experiments.
6.1 Experimental evaluation
The recent effort in extended summarization and its shared task of LongSumm (Chandrasekaran et al., 2020) used average ROUGE (F1) to rank the participating systems, in addition to commonly-used ROUGE-N scores. Table 2 shows the performance of the participated systems on the blind test set. As shown, BERTSUMEXTMULTI model outperforms other models by a large margin (i.e., with relative improvements of  $6\%$  and  $3\%$  on ROUGE-1 and average ROUGE(F1), respectively); hence, we use the best-performing in terms of F1 (i.e., BERTSUMEXTMULTI model) in our experiments.
Tables. 1 presents our results on the test sets of arXiv-Long and PubMed-Long datasets, respectively. As observed, our model statistically significantly outperforms the state-of-the-art systems on both datasets across most of the ROUGE vari