SlowGuess/NAACL_2022_OCR huggingface dataset:
ants, except ROUGE-L on PubMed-Long. The improvements gained by our model validates our hypothesis that incorporating the salient introductory sentence representations into the extractive summarizer yields a promising improvement. Two nonneural models (i.e., LSA and LEXRANK) underperform the neural models, as expected. Comparing the abstractive model (i.e., BART) with extractive neural ones (i.e., BERTSUMEXT and BERTSUMEXTMULTI), we see that while there is relatively a smaller gap in terms of ROUGE-1, the gap is larger for ROUGE-2, and ROUGE-L. Interestingly, in the case of BART, we found that generating extended summaries is rather challenging for abstractive summarizers. Current abstractive summarizers including BART have difficulty in abstracting very detailed information, such as numbers, and quantities, which hurts the faithfulness of the generated summaries to the source. This behavior has a detrimental effect, specifically, on ROUGE-2 and ROUGE-L as their high correlation with human judgments in terms of faithfulness has been shown (Pagnoni et al., 2021). Comparing the extractive BERTSUMEXT and BERTSUMEXTMULTI models, while BERTSUMMULTIEXT is expected to outperform BERTSUMEXT, it is observed that they perform almost similarly, with small (i.e., insignificant) improved metrics. This might be due to the fact that BERTSUMEXTMULTI works out-of-the-box when a handful amount of sentences are sampled from diverse sections to form the oracle summary as also reported by its authors. However, when labeling oracle sentences in our framework (i.e., Intro-guided labeling), there is no guarantee that the final set of oracle sentences are labeled from diverse sections. Overall, our model achieves about  $1.4\%$ ,  $2.4\%$ ,  $3.5\%$  (arXiv-Long), and  $1.0\%$ ,  $2.5\%$ ,  $1.3\%$  (PubMed-Long) improvements across ROUGE score variants; and  $2.2\%$  (arXiv-Long),  $1.4\%$  (PubMed-Long) improvements over F1, compared to the neural baselines (i.e., BERTSUMEXT and BERTSUMEXTMULTI). While comparing our model with BERTSUMEXT-INTRO, we see the vital effect of adding second encoder at finding supplementary sentences across non-introductory sections, where our model gains relative improvements of  $9.62\%-26.26\%-16.09\%$  and  $9.40\%-5.27\%-9.99\%$  for ROUGE-1, ROUGE-2, ROUGE-L on arXiv-Long and PubMed-Long, respectively. In fact, the sentences that are picked as summary from the in
$^{7}$ https://github.com/Georgetown-IR-Lab/TSTRSum
<sup>8</sup>We used mean of ROUGE-2 and ROUGE-L.
<sup>9</sup>We assumed that non-introductory sentences occur in sections other than the first section.
Model	arXiv-Long	PubMed-Long
R1(%)	R2(%)	RL(%)	F1 (%)	R1(%)	R2(%)	RL(%)	F1 (%)
ORACLE	53.35	24.40	23.65	33.80	52.11	23.41	25.42	33.65
BERTSUMEXT-INTRO	44.88	15.99	19.14	26.25	45.08	20.08	21.52	28.89
LSA	43.23	13.47	17.50	24.73	44.47	15.38	19.17	26.34
LEXRANK	43.73	15.01	18.62	25.41	48.63	20.37	22.49	30.50
BERTSUMEXT	48.42	19.71	21.47	29.87	48.82	20.89	23.37	31.03
BERTSUMEXTMULTI	48.52	19.66	21.42	29.87	48.85	20.71	23.29	30.95
BART	48.12	15.30	20.80	28.07	48.32	17.33	21.42	29.87
TSTR (Ours)	49.20*	20.19*	22.22*	30.54	49.32*	21.41*	23.67	31.47
Table 1: ROUGE (F1) results of the baseline models and our model on the test sets of the extended summarization datasets (arXiv-Long, and PubMed-Long). * shows the statistical significance (paired t-test,  $p < {0.05}$  ).
R1	R2	RL	F1(%)
Summaformers (2020)	49.38	16.86	21.38	29.21
IIITBH-IIITP (2020)	49.03	15.74	20.46	28.41
Auth-Team (2020)	50.11	15.37	19.59	28.36
CIST_BUPT (2020)	48.99	15.06	20.13	28.06
BERTSUMEXTMULTI (2021)	53.11	16.77	20.34	30.07
Table 2: ROUGE (F1) results of different systems on the blind test set of LongSumm dataset containing 22 abstractive summaries.
Introduction section are not comprehensive as such they are clues to the main points of the paper. The other important sentences are picked from the supplementary parts (i.e., non-introductory) of the paper.
6.2 Human evaluation
While our model statistically significantly improves upon the state-of-the-art baselines in terms of ROUGE scores, a few works have reported the low correlation of ROUGE with human judgments (Liu and Liu, 2008; Cohan and Goharian, 2016; Fabbri et al., 2021). In order to provide insights into why and how our model outperforms the best-performing baselines, we perform a manual analysis of our system's generated summaries, BERT-SUMEXT's, and BERTSUMEXTMULTI's. For the sake of evaluation, two annotators were asked to manually evaluate two sets of 40 papers' ground-truth abstracts (40 for arXiv-Long, and 40 for PubMed-Long) with their generated extended summaries (baselines' and ours) to gain insights into qualities of each model. Annotators were Electrical Engineering and Computer Science PhD students and familiar with principles of reading scientific
papers. Samples were randomly selected from the test set, one from each 40 evenly-spaced bins sorted by the difference of ROUGE-L between two experimented systems.
The evaluations were performed according to two metrics: (1) Cohesion: whether the ordering of sentences in summary is cohesive, namely sentences entail each other. (2) Completeness: whether the summary covers all salient information provided in the ground-truth summary. To prevent bias in selecting summaries, the ordering of system-generated summaries were shuffled such that it could not be guessed by the annotators. Annotators were asked to specify if the first system-generated summary wins/loses or ties with the second system-generated summary in terms of qualitative metrics. It has to be mentioned that since our model is purely extractive, it does not introduce any fact that is unfaithful to the source.
Our human evaluation results along with Cohen's kappa (Cohen, 1960) inter-rater agreements are shown in Table 3 (agr. column). As shown, our system's generated summaries improve completeness and cohesion in over  $40\%$  for most of the cases (6 out of 8 for win cases  $^{10}$ ). Specifically, when comparing with BERTSUMEXT, we see that  $68\%$ ,  $80\%$  (arXiv-Long); and  $60\%$ ,  $66\%$  (PubMed-Long) of sampled summaries are at least as good as or better than the corresponding baseline's generated summaries in terms of cohesion and completeness, respectively. Overall, across two metrics for BERTSUMEXT and BERTSUMEXT-MULTI, we gain relative improvements over the baselines:  $25.6\%$ ,  $19.0\%$  (cohesion), and  $56.5\%$ ,
<sup>10</sup>Win cases are the ones in which our system wins the baseline(s) in terms of cohesion/completeness.
Metric	Win	Tie	Lose	agr.
Our Model vs. BERTSUMEXT baseline
Cohesion	43%	25%	32%	46.5%
Completeness	46%	34%	20%	48.9%
Our Model vs. BERTSUMEXTMULTI baseline
Cohesion	42%	24%	34%	47.2%
Completeness	45%	32%	24%	49.1%
(a)
Metric	Win	Tie	Lose	agr.
Our Model vs. BERTSUMEXT baseline
Cohesion	39%	21%	30%	52.1%
Completeness	47%	19%	34%	51.3%
Our Model vs. BERTSUMEXTMULTI baseline
Cohesion	37%	21%	32%	48.2%
Completeness	41%	17%	32%	46.3%
Table 3: Results of human evaluations over 40 papers sampled from (a) arXiv-Long's, and (b) PubMed-Long's test set. agr. shows inter-rater agreement.
[Introductory] The objective of the work presented here is to study the mechanism of radiative line driving and the corresponding properties of the winds of possible generations of very massive stars at extremely low metallicities and to investigate the principal influence of these winds on ionizing fluxes and observable ultraviolet spectra.  $[s_1]$  The basic new element of this approach, needed in the domain of extremely low metallicity, is the introduction of depth dependent force multipliers representing the radiative line acceleration.  $[s_2]$  [...] Because of the depth dependent force multipliers a new formulation of the critical point equations is developed and a new iterative solution algorithm for the complete stellar wind problem is introduced (section 4).  $[s_3]$
[Non-introductory] In this section we develop a fast algorithm to calculate stellar wind structures and mass - loss rates from the equation of motion (eq.[eom1]) using a radiative line acceleration parametrized in the form of eq.[fmp3].  $[s_4]$  After the new concept to calculate stellar wind structures with variable force multipliers has been introduced and tested by comparing with the observed wind properties,  $[s_5]$  The purpose of this first study is to provide an estimate about the strengths of stellar winds at very low metallicity for very massive hot stars in a mass range roughly between 100 to  $300\mathrm{m}$  @xmath3.  $[s_6]$  With our new approach to describe line driven stellar winds at extremely low metallicity we were able to make first predictions of stellar wind properties, ionizing fluxes and synthetic spectra of a possible population of very massive stars in this range of metallicity.  $[s_7]$  [...] We also calculated synthetic spectra and were able to present for the first time predictions of uv spectra of very massive stars at extremely low metallicities.  $[s_8]$  We learned that the presence of stellar winds leads to observable broad spectral line features, which might be used for spectral diagnostics, should such an extreme stellar population be detected at high redshift.  $[s_9]$  [...]
(a)
(b)
Figure 4: (a) Our system's generated summary, (b) Sentence graph visualization of our system's generated summary. Green and gray nodes are introductory and non-introductory sentences, respectively. Edge thickness denotes the ROUGE score strength between pair of sentences. Parts, from which sentences are sampled, are shown inside brackets. The summary is truncated due to space limitations. Ground-truth summary-worthy sentences are underlined, and colored spans show pointers from introductory to non-introductory sentences.
$46.7\%$  (completeness) on arXiv-Long; and  $23.1\%$ ,  $13.5\%$  (cohesion), and  $27.7\%$ ,  $21.9\%$  (completeness) on PubMed-Long. $^{11}$  These improvements, qualitatively evaluated by the human annotators, show the promising capability of our purposes model in generating improved extended summaries which are more preferable than the baselines'. We observe a similar improvement trend when comparing our summaries with BERTSUMEXTMULTI, where  $66\%$ ,  $77\%$  (arXiv-Long); and  $58\%$ ,  $58\%$  (PubMed-Long) of our summaries are as good as or better than the baseline's in terms of cohesion and completeness. Looking at the Cohen's inter-rater agreement, the correlation scores fall into "moder
ate” agreement range according to the interpretation of Cohen's kappa range (McHugh, 2012).
6.3 Case study
Figure 4 (a) demonstrates an extended summary generated from a sample arXiv-Long paper by our model. The underlined sentences denote that the corresponding sentences are oracle (i.e., summary-worthy), the colored spans denote the pointers from introductory information to non-introductory information, and sentence numbers appear in brackets following each sentence. As shown, our system first identifies salient introductory sentences (i.e.,  $[s_1]$  and  $[s_3]$ ), and then augments them with important non-introductory sentences. Figure 4 (b) shows the ROUGE scores between pairs of intro
11Relative improvement of win rate over lose rate.
ductory and non-introductory sentences. The edge thickness signifies the strength of the ROUGE score between a pair of sentences. For example, introductory sentence  $[s_1]$  highly correlates with nonintroduc tory sentence  $[s_7]$  as it has a stronger edge  $(s_{1},s_{7})$  thickness. More specifically,  $[s_1]$  has mentions of "radiative line driving", "properties of the winds", "possible generations of very massive stars", and "ionizing fluxes" which maps to  $[s_7]$  with semantically similar mentions of "line driven stellar winds", "stellar wind properties", "possible generations of very massive stars", and "ioniz ing fluxes" 12.
To determine the limitations of our model, we further analyze our system's generated summaries and report three common defects, along with the percentage of these errors among underperformed cases. We found that (1) our end-to-end system's performance is highly dependent on the introductory sentence extraction task's performance (i.e., task  $\mathbf{t}_2$  in Figure 3) as identification of salient introductory sentences (i.e., oracle introductory sentences) sets up a firm ground to explore detailed sentences from the non-introductory parts of the paper. In other words, identification of non-salient introductory sentences leads to a drift in finding supplemental sentences from the non-introductory parts. Our model often underperforms when it cannot find important sentences from the introductory part (65%); (2) in underperformed cases, our model fails in selecting motivation, objective sentences from the introductory part, and only identifies the contribution sentences (i.e., describing paper's contributions), such that the final generated summary is composed of contribution sentences, rather than objective sentences. This observation hurts the system in cohesion and completeness (40%); and (3) as discussed, our model matches introductory sentences with sentences from non-introductory parts of the paper. Given that two sentences within a scientific paper might conceptually convey the exact same information, but are just paraphrased of each other, our model samples both to form the final summary as a high semantic correlation exists between them. This phenomenon leads to sampling two sentences that convey the same information
without providing more details; hence, information redundancy  $(35\%)$ .
In this work, we propose a novel approach to tackle the extended summary generation for scientific documents. Our model is built upon the fine-tuned contextualized language models for text summarization. Our method improves over strong and state-of-the-art summarization baselines by adding an auxiliary learning component for identifying salient introductory information of long documents, which are then used as pointers to guide the summarizer to pick summary-worthy sentences. The extensive intrinsic and human evaluations show the efficacy of our model in comparison with the state-of-the-art baselines, using two large scale extended summarization datasets. Our error analysis further paves the path for future research.
Abdullah Armagan. 2013. How to write an introduction section of a scientific article? Turkish journal of urology, 39 Suppl 1:8-9.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. *ArXiv*, abs/2004.05150.
Odellia Boni, Guy Feigenblat, Doron Cohen, Haggai Roitman, and David Konopnicki. 2020. A study of human summaries of scientific articles. *ArXiv*, abs/2002.03604.
Muthu Kumar Chandrasekaran, Guy Feigenblat, Edward Hovy, Abhilasha Ravichander, Michal Shmueli-Scheuer, and Anita de Waard. 2020. Overview and insights from the shared tasks at scholarly document processing 2020: CL-SciSumm, LaySumm and LongSumm. In Proceedings of the First Workshop on Scholarly Document Processing, pages 214-224, Online. Association for Computational Linguistics.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
Arman Cohan and Nazli Goharian. 2015. Scientific article summarization using citation-context and article's discourse structure. In Proceedings of the 2015
12The entire system-generated summaries are publicly available at https://github.com/Georgetown-IR-Lab/TSTRSum, including 40 human-evaluated cases.
Conference on Empirical Methods in Natural Language Processing, pages 390-400, Lisbon, Portugal. Association for Computational Linguistics.
Arman Cohan and Nazli Goharian. 2016. Revisiting summarization evaluation for scientific articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 806-813, PortoRoZ, Slovenia. European Language Resources Association (ELRA).
Arman Cohan and Nazli Goharian. 2018. Scientific document summarization via citation contextualization and scientific discourse. International Journal on Digital Libraries, 19(2):287-303.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37-46.
Peng Cui and Le Hu. 2021. Sliding selector network with dynamic memory for extractive summarization of long documents. In *NAACL*.
Peng Cui, Le Hu, and Yuanchao Liu. 2020. Enhancing extractive text summarization with topic-aware graph neural networks. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5360-5371, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res., 22:457-479.
A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Chong Feng, Fei Cai, Honghui Chen, and Maarten de Rijke. 2018. Attentive encoder-based extractive text summarization. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, pages 1499-1502. ACM.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Sayar Ghosh Roy, Nikhil Pinnaparaju, Risubh Jain, Manish Gupta, and Vasudeva Varma. 2020. Summaformers @ LaySumm 20, LongSumm 20. In Proceedings of the First Workshop on Scholarly Document Processing, pages 336-343, Online. Association for Computational Linguistics.
Alexios Gidiotis, Stefanos Stefanidis, and Grigorios Tsoumakas. 2020. AUTH @ CLSciSumm 20, LaySumm 20, LongSumm 20. In Proceedings of the First Workshop on Scholarly Document Processing, pages 251-260, Online. Association for Computational Linguistics.
Ruipeng Jia, Yanan Cao, Haichao Shi, Fang Fang, Yanbing Liu, and Jianlong Tan. 2020. Distilsum: Distilling the knowledge for extractive summarization. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 2069-2072. ACM.
Padma Rekha Jirge. 2017. Preparing and publishing a scientific manuscript. Journal of Human Reproductive Sciences, 10:3-9.
Darsh Kaushik, Abdullah Faiz Ur Rahman Khilji, Utkarsh Sinha, and Partha Pakray. 2021. CNLP-NITS @ LongSumm 2021: TextRank variant for generating long summaries. In Proceedings of the Second Workshop on Scholarly Document Processing.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Lei Li, Yang Xie, Wei Liu, Yinan Liu, Yafei Jiang, Siya Qi, and Xingyuan Li. 2020. CIST@CL-SciSumm 2020, LongSumm 2020: Automatic scientific document summarization. In Proceedings of the First Workshop on Scholarly Document Processing, pages 225–234, Online. Association for Computational Linguistics.
Feifan Liu and Yang Liu. 2008. Correlation between ROUGE and human evaluation of extractive meeting summaries. In Proceedings of ACL-08: HLT, Short Papers, pages 201-204, Columbus, Ohio. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
M. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia Medica, 22:276 - 282.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 3075-3081. AAAI Press.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. ArXiv, abs/2104.13346.
Vahed Qazvinian and Dragomir R. Radev. 2008. Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 689-696, Manchester, UK. Coling 2008 Organizing Committee.
Juan Ramirez-Orta and Evangelos E. Milios. 2021. Unsupervised document summarization using pretrained sentence embeddings and graph centrality. In SDP.
Saichethan Reddy, Naveen Saini, Sriparna Saha, and Pushpak Bhattacharyya. 2020. IIITBH-IITP@CLSciSumm20, CL-LaySumm20, LongSumm20. In Proceedings of the First Workshop on Scholarly Document Processing, Online. Association for Computational Linguistics.
T. Rohde, Xiaoxia Wu, and Yinhan Liu. 2021. Hierarchical learning for generation with long source sequences. ArXiv, abs/2104.07545.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Sajad Sotudeh, Arman Cohan, and Nazli Goharian. 2020. GUIR @ LongSumm 2020: Learning to generate long summaries from scientific documents. In Proceedings of the First Workshop on Scholarly Document Processing, pages 356-361, Online. Association for Computational Linguistics.
Sajad Sotudeh, Arman Cohan, and Nazli Goharian. 2021. On generating extended summaries of long documents. The AAAI-21 Workshop on Scientific Document Understanding (SDU).