SumSurvey: An Abstractive Dataset of Scientific Survey Papers for Long Document Summarization (2024)

Abstract: With the popularity of large language models (LLMs) and their ability to handle longer input documents, there is a growing need for highquality long document summarization datasets. Although many models already support 16k input, current lengths of summarization datasets are inadequate, and salient information is not evenly distributed. To bridge these gaps, we collect a new summarization dataset called SumSurvey, consisting of more than 18k scientific survey papers. With an average document length exceeding 12k and a quarter exceeding 16k, as well as the uniformity metric outperforming current mainstream long document summarization datasets, SumSurvey brings new challenges and expectations to both fine-tuned models and LLMs. The informativeness of summaries and the models supporting the evaluation of long document summarization warrant further attention. Automatic and human evaluation results on this abstractive dataset confirm this view. Our dataset and code are available at https://github.com/ Oswald1997/SumSurvey.

heading: 1 Introduction
text: As one of the major problems in natural language processing, summarization is about processing input documents and generating short texts that contain key information. Its main applications include extracting main events of news, mining opinions on social media, and analyzing comments on shopping websites (Nallapati et al., 2016; Bilal et al., 2022; Angelidis and Lapata, 2018). These tasks use extractive or abstractive models for summarization. The former refers to the extraction of key words or sentences from input to form a summary (Erkan and Radev, 2004; Rossiello et al., 2017), while the latter is to rewrite important information
*Corresponding author.
on the basis of understanding (Cho et al., 2022; Liu et al., 2022). Intuitively, abstractive tasks are more challenging, which require models to read the input document and present the most salient information within output length limits. In the process of improving and evaluating this capability of summarization models, abstractive datasets play an indispensable role.
In addition, the distribution form of salient information is also an indicator of quality for summarization datasets, because a more uniform distribution means that obvious layout bias cannot be exploited (Koh et al., 2022a). For some tasks, extracting only the first few sentences directly as the final summary can achieve good results, which is common in news domain (Ishikawa et al., 2001). Summarization models need to be able to extract key information distributed throughout the document and deal with relationship among different positions.
With the explosion of data volume and increase of computing resources, language models can gradually deal with longer documents (Beltagy et al., 2020; Guo et al., 2022). However, even with the ability to handle long text, it is still difficult to evaluate the performance of these models due to the different lengths of documents in public datasets.
9632
The above problems bring a requirement to summarization task: To construct a new summarization dataset with high abstractiveness and uniformity of salient information distribution, while lengths of documents are in line with the processing capacity of current language models.
We believe that collecting survey papers is a feasible approach, because such documents are usually long and enumerate many studies in a certain field in detail. Besides, the abstract section naturally summarizes the full paper. Compared with general papers, information in survey papers tend to be more evenly distributed. Thus, we propose a long document summarization dataset called SumSurvey consisting of scientific survey papers. Figure 1 shows a random summary example1 from SumSurvey, different colored parts indicate that they are generated from different sections, corresponding to contents from chapter I to chapter VI. It can be seen that each section is reflected in this summary, and is presented in a highly general discourse, which is exactly what we need.
We conduct a series of experiments using finetuned models (Beltagy et al., 2020; Phang et al., 2023; Guo et al., 2022) and large language models(Du et al., 2022; Zheng et al., 2023). Both automatic and human evaluation are conducted to explore models’ ability to handle long documents. Our main contributions are as follows:
• We propose a new long document summarization dataset called SumSurvey, consisting of scientific survey papers. To our knowledge, it is an English summarization dataset with the longest average document length compared with publicly available datasets, and has advantages in abstractiveness, distribution uniformity, summary non-redundancy and compression rate.
• We benckmark various fine-tuned models and LLMs. Both automatic and human evaluation are conducted to comprehensively evaluate the quality of summaries. We also investigate limitations of automatic evaluation on long document summarization.
• We discuss the impact and expectations of SumSurvey on the community. We hope SumSurvey will contribute to evaluating and further improving fine-tuned models on longer
1The random sample is from here
inputs, and also expect LLMs to pay more attention to informativeness and factuality. In addition, we hope to see more models that support automatic evaluation of long document summarization.

heading: 2 Related Work
text: Summarization Datasets Datasets are critical to improving and evaluating model performance. For summarization task, there exists many publicly available datasets in various domains. According to the number of source documents in each sample, they can be categorized into single-document (Grusky et al., 2018; Kornilova and Eidelman, 2019) and multi-document (Fabbri et al., 2019; Lu et al., 2020) summarization datasets. From the perspective of source document length, they can be classified into short and long document summarization dataset. Koh et al. (2022b) define datasets that cannot be directly read by pre-trained Transformers as long document datasets. Phang et al. (2023) extend the text length to 4096 tokens and considers it as long input. As many models are being extended to support longer input, currently widely used long document datasets (Cohan et al., 2018; Sharma et al., 2019; Kornilova and Eidelman, 2019) can not meet the demand, and many of them filtered out particularly long documents. Although average length of documents in some datasets is long (Huang et al., 2021), abstractiveness of their summaries is not high (See §3 for more details). In conclusion, there is a lack of long document summarization datasets with high quality, such as high abstractiveness and uniformity of salient information. Summarization Models Pre-trained language models are mainstream methods for summarization. Some generation models are task-generic and can be fine-tuned to fit summarization scenario (Lewis et al., 2020; Raffel et al., 2020). Other models focus on summarization, researchers design objective functions during pre-training to make models generate higher quality summaries (Zhang et al., 2020a; Xiao et al., 2022). Since these models are based on Transformer (Vaswani et al., 2017), they have quadratic time complexity so cannot adapt well to long document summarization, which has longer input and more topic coverage. The use of efficient attention mechanism can improve the efficiency of processing long documents. For example, Longformer utilizes sparse attention where win-
dow has gaps and adds global attention to adapt to different tasks (Beltagy et al., 2020). BigBird combines global, sliding and random attention to reduce quadratic dependency to linear (Zaheer et al., 2020). Zhang et al. (2021) explored three different strategies for long dialogue summarization, and more recent studies investigated the impact of different Transformer modules (Phang et al., 2023) and the use of large language models for long document summarization (Syed et al., 2023; Ravaut et al., 2023) or other tasks (Liu et al., 2023a). In many cases, models can only be evaluated with at most 8k input length limited by current datasets (Manakul and Gales, 2021; Koh et al., 2022b). Therefore, the proposal of our SumSurvey will alleviate this situation.

heading: 3 SumSurvey

heading: 3.1 Dataset Construction
text: We crawled all searchable papers on arxiv.org which have the word survey in their titles2. Abstract of each paper is extracted precisely because it can be matched directly from web page. As for main body, we downloaded PDF files and extracted the plain text, then we removed all contents including abstract section and before, and removed contents including references section and after. After cleaning and filtering, we obtain a total of 18,884 samples. The main body of each sample is used as input, and abstract is target. We split our dataset into train (15,108, 80%), validation (1,888, 10%), and test (1,888, 10%) subsets. More details about dataset construction and discussions on data quality are in Appendix A.1 and Appendix A.2.
The average input document length of SumSurvey is 12k, which exceeds existing summarization datasets. More than half of the documents are over 10k in length, in addition, a quarter of documents are longer than 16k, which is the maximum input length supported by many extended models, meaning that our dataset is well suited to evaluating their performance. Papers in SumSurvey spans from 1991 to 2023. Generally, there tends to be more papers in recent years than in the past. These papers come from a variety of subjects, top subjects are astrophysics, machine learning, computer vision and pattern recognition, artificial intelligence, cryptography and security. The subject distribution conforms to a long-tail distribution. More details
2Deadline is 19th May, 2023.
about distributions of lengths, years and fields are in Appendix A.3

heading: 3.2 Dataset Properties
text: In this section, we use four indicators to evaluate intrinsic characteristics of datasets. We choose five commonly used English long document datasets for comparison. PubMed and arXiv (Cohan et al., 2018) are from scientific papers. BigPatent (Sharma et al., 2019) consists of records of U.S. patent documents. BillSum (Kornilova and Eidelman, 2019) is a summarization dataset from U.S. Congressional and California state bills. GovReport (Huang et al., 2021) is a collection of reports published by U.S. Government Accountability Office and Congressional Research Service. Coverage measures the percentage of words in a summary that are part of an extractive fragment from the document. It was proposed by Grusky et al. (2018) and the calculation method is expressed as:
Coverage(A,B) = 1 |B| ∑
f∈F (A,B) |f | (1)
where A and B represent the document and its summary respectively. F (A,B) is the set that includes all extractive fragments. | · | is the length of a token sequence. Larger coverage value means more contents are copied directly from document when generating summary. Density is similar to coverage, where the sum of fragment lengths is changed to the sum of squares of lengths (Grusky et al., 2018):
Density(A,B) = 1 |B| ∑
f∈F (A,B) |f |2 (2)
If length of each fragment is short, the density value will be low, which means that if two summaries have the same coverage value, the one with lower density might have more variability, because its fragments are short rather than long and continuous. Redundancy is used to evaluate whether sentences in a summary are similar to each other (Bommasani and Cardie, 2020):
Redundancy(B) = mean (x,y)∈S×S,x̸=y Rl(x, y) (3)
In this formula S is sentence set of summary B, (x, y) is a sentence pair. Rl is ROUGE-L F1-score (Lin, 2004). Larger Rl means higher degree of
overlap in a sentence pair. Therefore, redundancy can be used to indicate the extent to which sentences in a summary are redundant. In general, a high-quality summary needs to be as concise as possible. Uniformity measures the degree to which salient information in a summary is evenly distributed throughout the document:
Uniformity(A,B) = − 1 n
m∑
i=1
pi log(pi) (4)
We divide source document A equally into m parts and calculate the probability of the salient word3 in the summary B falling into each of these parts. If different pi values are close, the final uniformity value will be high.
Table 1 shows coverage, density, redundancy and uniformity scores of several datasets, SumSurvey has the highest overall ranking. To be specific, BigPatent achieves highest scores on coverage and density, closely followed by SumSurvey, which means that less summary contents in these two datasets are extracted from documents, and each single extracted fragment is relatively short in length, thus the possibility of rearranging these fragments is high. We plot heatmaps in Appendix B.1 to show coverage and density more intuitively. Apart from coverage and density, we also calculate proportions of novel n-grams in summaries to further evaluate the abstractiveness of datasets, see Appendix B.2 for results.
SumSurvey has good redundancy score, this is consistent with the characteristics of abstracts in survey papers. In addition, SumSurvey achieves the highest uniformity score, which also benefits from the fact that these documents are derived from survey papers, because typically large amounts of contents scattered throughout the survey paper are about describing previous research and methods, besides, these contents are equally important, so they will be all reflected in the abstract.

heading: 4 Experiments
text: We conduct a series of experiments to evaluate performance of baseline models on SumSurvey.

heading: 4.1 Baselines
text: We use LED (Beltagy et al., 2020), PEGASUSX (Phang et al., 2023) and LongT5 (Guo et al.,
3Salient words are extracted by NLTK.
2022) as baselines. LED is based on Longformer, it combines a local windowed attention and a task motivated global attention. PEGASUS-X uses staggered block-local Transformer with global encoder tokens. LongT5 integrates attention ideas from ETC and adopts pre-training strategies from PEGASUS. These models support long input at most 16k tokens. More details about these baselines are described in Appendix C.1.
In addition, we evaluate large language models under zero-shot settings. We choose ChatGPT, ChatGLM3 (Du et al., 2022) and Vicuna (Zheng et al., 2023), which all have versions that support long inputs. As these models are all instructiontuned models, they are often capable of generalizing to unseen tasks (Longpre et al., 2023; Chung et al., 2024; Iyer et al., 2022).

heading: 4.2 Settings
text: For the pre-trained summarization models, we use led-large-16384 4 and long-t5-tglobal-base 5 for summarization of 16k input tokens, while pegasus-x-large6 is adopted for 10k only, because fine-tuning PEGASUS-X under 16k tokens with batch size of 1 requires more than 80GB of GPU memory, which is beyond the computing resources we have. Nevertheless, our length settings still exceed experiments in previous research. For zero-shot LLMs, we use gpt-3.5-turbo-16k7, chatglm3-6b-32k8 and vicuna-13b-v1.5-16k9 for implementation. We evaluate ChatGPT by OpenAI API, while the remaining LLMs are evaluated locally.
We use an NVIDIA A100 80GB PCIe GPU for experiments. Models are fine-tuned for 10 epochs, other parameters vary depending on different experiment settings. Beam search is adopted in inference phase and we set beam size to be 5. Implementation details are in Appendix C.2.
4https://huggingface.co/allenai/ led-large-16384
5https://huggingface.co/google/ long-t5-tglobal-base
6https://huggingface.co/google/ pegasus-x-large
7https://platform.openai.com/docs/models/ gpt-3-5-turbo
8https://huggingface.co/THUDM/chatglm3-6b-32k 9https://huggingface.co/lmsys/vicuna-13b-v1.
5-16k

heading: 5 Results

heading: 5.1 Automatic Evaluation

heading: 5.1.1 Reference-Based Evaluation
text: Typically, reference-based methods evaluate summarization models by measuring the similarity between generated summaries and the references. Several evaluation methods are used in this section: ROUGE We use F1-score of ROUGE-1, ROUGE2 and ROUGE-L10, taking into account the completeness, readability and order of summary. BERTScore BERTScore (Zhang et al., 2020b) computes a similarity score for each token in the candidate summary with each token in the reference summary. The similarity score is calculated using contextual embeddings rather than exact matches like ROUGE. In implementation, robertalarge is used to represent embeddings (Liu et al., 2019). UniEval-Relevance UniEval (Zhong et al., 2022) reconstructs generation evaluation into a Boolean Question Answering (QA) task. By providing a summary and a reference, it can calculate a relevance score, indicating whether the summary contains only the important information. ChatGPT and Vicuna We use powerful LLMs to evaluate summary quality. We prompt these models (Appendix C.3) to rate the similarity between the summary and the reference on a scale from 1 to 5.
See Table 2 for results. Large language models overall scored lower on ROUGE compared to finetuned models, which can be expected as summaries generated by LLMs tend to be more abstractive. All BERTScore results are quite similar, indicating that this method might not be well-suited for
10ROUGE-1.5.5 is used for evaluation.
evaluating summarization performance in SumSurvey. In contrast, scores generated by UniEval are more discriminative. Unlike ROUGE, the scores of large language models are consistently higher than fine-tuned models. Among them, ChatGLM3 performs the best, while PEGASUS-X scores notably lower than other models. Evaluation results by both LLMs indicate that the quality of summaries generated by Vicuna is significantly lower than those generated by ChatGPT and ChatGLM, and even lower than fine-tuned models. It seems that language models tend to favor summaries generated by themselves, for example, ChatGPT indicates its summaries are similar in score to those generated by ChatGLM3, while Vicuna suggests there is a significant difference in scores between them.

heading: 5.1.2 Reference-Free Evaluation
text: In this section, we use reference-free metrics to evaluate linguistic quality and abstractiveness of summaries: UniEval-Fluency By providing a question and a paragraph, the model can evaluate whether the summary is fluent. ChatGPT and Vicuna LLMs will assign three scores ranging from 1 to 5 for each summary, corresponding to its grammatical accuracy, coherence, and referential clarity. Novel N-Gram This metric is used to measure the abstractiveness of summaries.
See Table 3 for results of linguistic quality. Text generated by large language models exhibits higher linguistic quality, which is one of the main advantages of LLMs. UniEval considers LLMs to be superior overall to fine-tuned models but fails to distinguish between different models within each category. Evaluation results based on LLMs indicate that summaries generated by Vicuna are lower in all three metrics compared to the other two large
language models as well as LED. Table 4 shows the proportion of novel n-grams in summaries. Since all three LLMs have not been fine-tuned on SumSurvey, the style of their generated text comes from previous data, therefore they reach a high abstractiveness. Among them, ChatGLM3 exhibits the highest abstractiveness, with its generated summaries being comparable to humanwritten summaries. PEGASUS-X has lowest proportion of novel n-grams, indicating that it tends to generate summaries on SumSurvey in an extractive way.

heading: 5.1.3 Summary
text: The advantages of automatic evaluation are convenience and speed. However, the results presented by different evaluation methods can sometimes vary greatly, especially adopting reference-based methods. Some reference-free methods use the source document as input to evaluate factuality of generated summaries, for example, UniEvalConsistency, FactCC (Kryscinski et al., 2020) and QuestEval (Scialom et al., 2021). However, most of these methods are based on BERT or T5, and are inefficient in handling long inputs, making it challenging to automatically evaluate factuality in long document summarization. We also attempted to use LLMs that support long inputs to evaluate the informativeness and factuality of summaries, but instability issues often arose. Exploring suitable evaluation methods for long document summarization is a future research direction.

heading: 5.2 Human Evaluation
text: The summaries generated by fine-tuned models depend on learning the mapping relationship between source documents and references during finetuning, while the summaries generated by large language models are based on the knowledge learned in previous phases and the designed prompts. Due
to the different paradigms, automated evaluation methods may not accurately assess summary quality. For example, ROUGE scores focus on the overlap between generated summaries and references. Fine-tuned models often generate summaries with lower abstractiveness compared to LLMs, which may result in higher scores for fine-tuned models. We aim to alleviate this bias through human evaluation and gain a comprehensive understanding of summarization performance of different models on SumSurvey.
Three graduate students serving as annotators rated summaries generated by different models on four indicators: fluency, coherence, nonredundancy, and informativeness. Considering the workload, we randomly selected 50 samples for evaluation. We formulated human evaluation guidelines to clarify the meanings represented by different indicators (Appendix D).
See Table 5 for human evaluation results. We computed Kendall’s coefficient of concordance (Kendall-W) for inter-annotator agreement11, annotators reached a moderate agreement on fluency and coherence, and reached a substantial agreement on non-redundancy and informativeness. Three large language models scored higher overall in fluency compared to fine-tuned models, consistent with the results in Table 3. However, in terms of coherence, ChatGLM3 performed poorly. Upon inspection, we discovered that summaries generated by ChatGLM3 often lack a cohesive paragraph structure and are instead divided into numerous points, resulting in annotators perceiving them as incoherent. An important point to note is that automatic evaluation methods cannot distinguish fine-grained linguistic quality indicators, whereas human evaluation can differentiate fluency and coherence. In
11Kendall-W for fluency, coherence, non-redundancy, informativeness are 0.54, 0.58, 0.68, 0.62 respectively.
terms of non-redundancy, LED outperformed ChatGPT and Vicuna, we find that LED-generated summaries are very concise, leading to high score in non-redundancy. As for informativeness, there is no significant distinction between models, as large language models do not demonstrate superiority. Vicuna performs the lowest on this indicator, we later find that summaries generated by Vicuna are relatively short, hence unable to contain much information.
Comparing the results of automatic evaluation and human evaluation, we find that automatic evaluation methods perform relatively well in linguistic quality metrics, closely aligning with human preference, but show significant differences in contentrelated metrics. This is because many evaluation methods are based on language models, thus they can more accurately evaluate linguistic quality.

heading: 6 Discussion

heading: 6.1 Is SumSurvey a challenging dataset?
text: The average length of samples in SumSurvey exceeds 12k, with a quarter of samples exceeding 16k in length. Due to high uniformity of SumSurvey, models find it challenging to utilize positional bias
for summarization. Additionally, given the specificity of scientific papers, many domain-specific terms need to be retained when generating summaries. However, our dataset still demonstrates high abstractiveness, necessitating summarization models to possess not only high abstractive capabilities but also the ability to identify and preserve domain-specific terms. This significantly increases the challenges associated with SumSurvey.
Appendix E shows ROUGE scores of some baselines on arXiv and GovReport. Results on SumSurvey are significantly worse than on the other two datasets, especially than on GovReport, which has the lowest coverage and density scores according to Table 1. However, ROUGE scores in Table 2 are still within a reasonable range, documents in SumSurvey are not unsummarizable, they just require models to have better abilities to understand long and complicated text.
We conduct another set of experiments. We use two selection strategies of tokens in training, in addition to the normal way of selecting tokens in a natural order, there is another option called oracle. The oracle selector is implemented by greedily searching sentences that achieve maximum ROUGE-2 Recall till input length limit is reached (Manakul and Gales, 2021). See Table 6 for results. We observe that whether oracle selector is used has little
effect on results. When sentences are selected by oracle selector, the mapping between input and output is more natural during training, and it is easier for model to learn how to extract and utilize key features, thereby improving its inference ability. However, the difference between the oracle and normal way on SumSurvey is not obvious, indicating mapping abilities learned by model under these two settings are similar, which means distributions of two types of data are also similar and proves high uniformity and high challenging of SumSurvey.

heading: 6.2 What is the impact of SumSurvey on fine-tuned models?
text: Although many summarization models already support 16k long inputs, they have been limited by previous datasets to only test the performance with inputs of up to 8k in length. With SumSurvey, it is now possible to expand on previous experiments. We use BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020a) as base models of LED and PEGASUS-X, and obtained ROUGE scores of different models under varying input lengths.
See Table 6 for results, scores increase with length in most cases, indicating that if only basic models are used by truncating the input, it will lead to information loss. Therefore, it is necessary to use summarization models supporting long document in many scenarios. Besides, when the token length increases from 8k, there is still a significant improvement in results, especially for LED which improves greatly from 8k to 16k. It means that the latter part of SumSurvey still contains a lot of information. More than a quarter of documents in SumSurvey have more than 16k tokens, while current models can not process these documents well, so there is still room for research in the field of long document summarization models.
We hope that our proposed SumSurvey dataset will further evaluate the performance of summarization models on longer inputs and enhance summarization capability through fine-tuning. Additionally, we hope to see more models supporting longer inputs to handle samples in SumSurvey that exceed 16k in length.

heading: 6.3 What are the expectations for LLMs?
text: Summaries generated by LLMs tend to be preferred by humans, for they have fewer grammatical errors and are more fluent and coherent. But these models have a tendency to focus on linguistic aspects but struggle to ensure fidelity to the factual information
and alignment with the original source, so it may overfit unconstrained human evaluation, which is affected by annotators’ prior, input-agnostic preferences (Atri et al., 2023; Liu et al., 2023b). These findings are consistent with the results in Table 5. In terms of informativeness, LLMs lack an advantage over fine-tuned models. We hope that long survey papers containing rich information can be used to improve information extraction and summarization capabilities of LLMs. After all, while generating summaries that align with human preferences is important, it is also necessary for summaries to contain rich and factual information.
Another expectation for LLMs is their ability to evaluate informativeness and factuality of generated summaries, which requires inputting both the original document and the summary for evaluation. Due to the length of SumSurvey, some models we have tested are unable to stably evaluate informativeness and factuality. We hope to see more models that support automatic evaluation of long document summarization.

heading: 6.4 How does SumSurvey help in creating models supporting inputs exceeding 16k?
text: For models incapable of handling inputs exceeding 16k tokens, SumSurvey encourages further research into "extract then abstract" and "divide and conquer" approaches. Due to the high uniformity of our dataset, the "extract then abstract" approach tends to lose significant information, resulting in an insignificant performance improvement (as shown in Table 6). Concerning the "divide and conquer" approaches, the even distribution of information makes such methods feasible. We anticipate the emergence of better hierarchical models and feature fusion models.
For the creation of models supporting inputs exceeding 16k tokens, SumSurvey encourages the development of more efficient attention mechanisms. Since these models can access the entire document, better summarization performance can be achieved theoretically. However, this necessitates models with both long-context modeling capabilities and efficient processing techniques, which poses greater challenges.

heading: 7 Conclusion
text: We propose a new long document summarization dataset SumSurvey consisting of scientific survey papers. The average length of documents in SumSurvey is longer than publicly available summarization dataset, and it has higher abstractiveness. In addition, salient information is more evenly distributed throughout documents. By benchmarking baseline models using automatic and human evaluation, we have a comprehensive view of how these models perform on SumSurvey. We hope SumSurvey will contribute to evaluating and further improving fine-tuned models on longer inputs, and also expect LLMs to pay more attention to informativeness and factuality. Moreover, we look forward to more comprehensive automatic evaluation models supporting long documents.
Limitations
Upon examining the metadata, it was confirmed that over two-thirds of the documents from SumSurvey contain publication information. For those lacking such information, a manual inspection of a subset revealed that the vast majority of samples actually originate from published or peer-reviewed papers, with only a negligible fraction being unpublished, which is insufficient to significantly affect
the overall quality. However, due to the dataset’s diverse range of fields, including many that are unfamiliar to us, manually checking the text quality of all documents presents a considerable challenge.
We did not benchmark all baselines because: 1) limited by computation power, some models cannot be fine-tuned on a single NVIDIA A100 with 80GB GPU memory, such as pegasus-x-large and longt5-tglobal-large with 16k tokens; 2) some models are not publicly available at the moment (Rohde et al., 2021; Pang et al., 2023); 3) some large language models like LongAlpaca (Chen et al., 2023) exhibited instability on SumSurvey, often failing to generate outputs properly.
Documents in SumSurvey lack language diversity. The fact that English is the dominant language in academia makes it a single-language dataset.
Ethics Statement
Data collection approval was received from an ethics review board. Remuneration we paid to the annotators is above the average salary level in the area where the annotators are located. Since documents in our dataset are scientific papers, it is unlikely that offensive contents are included. However, not all papers in arXiv have been published, so they may contain unacceptable conclusions or non-original ideas. All codes and data used in this paper comply with the license for use.

heading: Acknowledgments
text: This work is supported by Youth Innovation Promotion Association CAS (No.2021155).

heading: A Dataset details
text: A.1 Construction Details
The data we have collected falls into two categories: metadata extracted directly from websites and downloaded PDF files. Regarding metadata, it contains a wealth of information, including authors, abstracts, subjects, initial and latest submission times, acceptance and publication information, and more. For PDF files, pdfminer12 and RE rules were utilized to extract the plain text and split it into a list of tuples, where each tuple contains a section header and the corresponding text. We excluded the abstract section and content before it, as well as the reference section and content after it. To enhance the accuracy of abstract matching and removal, we employed BERT-based Sentence Transformers13. This allowed us to calculate the similarity between the abstract identified using RE rules and the actual abstract from metadata. Subsequently, we filtered out certain outlier data based on this similarity measure. In this above process, plain text of tables was retained while figures were removed, but figure captions were preserved for summarization models to utilize this information. We kept the original citation format of the source document, but the reference section was removed.
A.2 Dataset Quality
In this section, we discuss whether papers in SumSurvey are of high quality and whether SumSurvey is a different dataset.
Regarding the former, both the "Journal ref" field and the "Comments" field in the extracted metadata contain information about the paper’s acceptance and publication information. By examining the information, we observed that more than two-thirds of papers contain information regarding publication, and most papers without such information were, in fact, published. Therefore, we confirm that the vast majority of papers in the dataset are high-quality, published survey papers.
As for the latter, we compared SumSurvey with the existing arXiv dataset for they are both scientific paper datasets. We examined 6440 samples from the test set of arXiv dataset, and used arXiv API14 to obtain additional information not provided by the arXiv dataset. We found that papers with
12https://github.com/euske/pdfminer 13https://github.com/UKPLab/
sentence-transformers 14https://info.arxiv.org/help/api/basics.html
"survey" in their titles account for approximately one percent. Based on this proportion, we estimate that there are no more than 3000 survey papers in the entire arXiv dataset, equivalent to approximately 15% of SumSurvey in terms of sample size. It is worth noting that papers in the old arXiv dataset are all dated before 2018. Additionally, we calculated that the average length of survey papers in arXiv dataset is only 6k, approximately half of SumSurvey, as the authors of the arXiv dataset have filtered out excessively long papers.
A.3 Dataset Distributions
Length Distribution Figure 2 is length distribution of documents in SumSurvey. Table 7 shows length statistics in different datasets. The input document length of our SumSurvey exceeds other datasets, and summary length is in a reasonable range. Year Distribution See Figure 3 for year distribution of papers in SumSurvey. The more recent the year, the greater the number of papers. This indi-
cates that the data in our dataset is relatively recent. Field Distribution There are over 100 types of tags in total, and each sample typically has multiple tags. Representative fields include Astrophysics, Machine Learning, Computer Vision and Pattern Recognition, Artificial Intelligence, Cryptography and Security. In addition , there are other fields such as Signal Processing, Methodology, Algebraic Geometry, General Finance, and more. The field distribution conforms to a long-tail distribution.
B Intrinsic Characteristics
B.1 Heatmaps
Heatmaps in Figure 4 show coverage and density of different datasets. BillSum and GovReport perform the worst, while BigPatent is the best. Note that the average document length of BigPatent is much smaller than SumSurvey, making it less necessary for humans to extract many contents directly from the document when generating the summary, because of the reduced workload and the fact that the summary and corresponding short document will be too similar if lots of contents are copied (see scores of GovReport for example). In addition, scope of its heatmap is too narrow, indicating that writing styles of these summaries are highly consistent. PubMed, arXiv and SumSurvey are all consists of scientific papers, while the heatmap of SumSurvey is more regular in shape, proving that papers we collected are indeed in a same category, which is survey. As for PubMed and arXiv, particularly long documents have been filtered out (Cohan et al., 2018), so it is likely that many survey papers are lost.
B.2 Novel N-grams
The proportion of novel n-grams in a summary reflects the degree of abstractiveness of this sample. We calculate percentages of uni-, bi-, tri-, and 4-grams in summaries for long document summa-
rization datasets, we also introduce two more longinput datasets SummScreen (Chen et al., 2022) and SQuALITY (Wang et al., 2022), see Table 8 for results.
As shown in Table 8, SumSurvey has higher abstractiveness than arXiv, PubMed, BillSum and GovReport, especially on tri-grams and 4-grams, this proves again that SumSurvey has good density scores (see Table 1). SummScreen and SQuALITY have an advantage in abstractiveness as their data is from TV programs or stories, making the text filled with colloquialism and lack technical
terms. So, there is a broader range of word choices when writing abstracts. Regarding SumSurvey, the abundance of technical terms naturally puts it at a disadvantage in abstractiveness. Small data size of SQuALITY makes it difficult to use for fine-tuning models to improve long document summarization performance. Besides, the average length of both datasets is only around 5k-6k and they cannot be used to evaluate the ability of summarization models to handle longer inputs.

heading: C Experiment Details
text: C.1 Baselines Description
BART (Lewis et al., 2020) is a denoising autoencoder for pre-training sequence-to-sequence models. By destroying and then reconstructing text, BART has the flexibility to process raw text and learn to reconstruct it effectively. The fine-tuned BART performs well on summarization task. LED (Beltagy et al., 2020) is based on Longformer, which replaces the standard self-attention in Transformer with combination of a local windowed attention and a task motivated global attention. A token with a local attention attends to its context tokens while a global attention token attends to all tokens across the sequence. LED is a Longformer variant supporting long input up to 16k tokens, and its parameters are initialized from BART. PEGASUS (Zhang et al., 2020a) uses a new objective called Gap Sentences Generation (GSG) during pre-training designed specifically for summarization task. The authors select sentences that are important to document and mask them, then train PEGASUS to generate these sentences.
PEGASUS-X (Phang et al., 2023) is an extension of the PEGASUS model supporting long input at most 16k tokens. The authors design staggered block-local Transformer with global encoder tokens, where a block-local attention token can only attend to other tokens within the same block and the block allocation across alternating layers is staggered. Additionally, long documents are included during pre-training to improve downstream summarization performance. LongT5 (Guo et al., 2022) integrates attention ideas from ETC, and adopt pre-training strategies from PEGASUS into the scalable T5 architecture. It uses a new attention mechanism called Transient Global (TGlobal), which mimics ETC’s local/global attention mechanism, but without requiring additional side-inputs.
C.2 Implementation Details
BART We use fairseq (Ott et al., 2019) for implementation with Python version of 3.8 and PyTorch version of 2.0.1. Learning rate is set to 1e-4. Batch size is set to 2 with update frequency of 8. Other parameters like label smoothing, dropout, attention dropout, weight decay, clip norm are consistent with the official example15. LED We use Transformers (Wolf et al., 2020) as framework with version of 4.31.0. We follow this fine-tuning notebook16 with learning rate of 1e-4. PEGASUS Transformers is used as framework for implementation and pytorch-lightning17 version is 1.0.4. We set learning rate to 1e-5 and batch size to 6. PEGASUS-X It is implemented on Flax18 with version of 0.6.11 and JAX19 version of 0.4.13. Learning rate is set to 1e-4. Batch size are 2, 1, 1 when maximum input length are 4k, 8k, 10k respectively. LongT5 We use Transformers for implementation. Learning rate is set to 5e-4, and batch size is 1 with gradient accumulation step of 32. ChatGPT We use OpenAI API20 for implementation with prompt of "Please summarize the following contents with no more than 256 words: ", and
15https://github.com/facebookresearch/fairseq/ blob/main/examples/bart/README.summarization.md
16https://colab.research.google.com/drive/ 12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v
17https://github.com/Lightning-AI/ pytorch-lightning
18https://github.com/google/flax 19https://github.com/google/jax 20https://platform.openai.com
refer to official use case21 for parameter settings. ChatGLM3 We use Transformer for implementation. The prompt is the same as used by ChatGPT. Vicuna We use FastChat22 as evaluation framework. The prompt is the same as used by ChatGPT.
For LLMs, experiments are under zero-shot settings. While for other baselines, the fine-tuning time varies from a few hours to two days.
C.3 LLMs for Evaluation
We use ChatGPT and Vicuna to evaluate summary quality. Prompt used in §5.1.1 is "I will provide you with five different summaries of the same document generated by different models, as well as a manually generated standard summary. Please rate each of the five model-generated summaries on a scale of one to five in terms of similarity to the standard summary.". And prompt used in §5.1.2 is "I will provide you with five different summaries of the same document generated by different models. Please rate each of the five model-generated summaries on a scale of one to five in terms of grammatical accuracy, coherence, and clarity of reference. "

heading: D Human Evaluation Guidelines
text: Three annotators23 scores summaries independently, they need to complete 50 subtasks, each of which consists of source document and five summaries generated by LED, LongT5, ChatGPT, ChatGLM3 and Vicuna, respectively. All summaries are lowercased and tokenized, making it impossible to use these features to determine which summaries are likely generated by the same model. We have developed a guideline for annotators, see Fig 5.

heading: E Results on other datasets
text: We show some results of baselines on other datasets in Table 9. On the one hand, limited by these datasets, experiments cannot be conducted with longer input length; on the other hand, scores on these datasets are higher than those on SumSurvey, indicating strong challenges of our dataset. Among them, scores of baselines on GovReport are particularly high, because GovReport has the lowest abstractiveness (see Table 1 and Table 8), therefore,
21https://platform.openai.com/docs/ api-reference/chat/create
22https://github.com/lm-sys/FastChat 23Graduate students, two of whom are non-native English
speakers and one is a native English speaker.
models tend to generate summaries in an extractive way.

heading: F Case Study
text: After checking the summaries generated by baselines, we find that even as abstractive fine-tuned models, they tend to generate summaries in an extractive way when facing long document summarization task. This may be because documents in SumSurvey are too long for models to understandthen-reword, so summaries are generated by identifying salient sentences.
See Figure 6 and Figure 7 for examples. The summary generated by LED is more precise, while PEGASUS-X and LongT5 generates some examples, resulting in a bit of verbosity. The summary generated by ChatGPT has higher abstractiveness and contains some information that could easily be ignored. The summary generated by ChatGLM3 is not a complete paragraphin, thus achieving a low coherence score in human evaluation. Typically, even when given length constraint in the prompt, the summary generated by Vicuna is still relatively short, resulting in low informativeness.

heading: G Software and Licenses
text: Data and codes used in this paper are:
• arXiv, Misc24
• BERTScore, MIT25
• Datasets, Apache-2.026
• fariseq, MIT27
• Flax, Apache-2.028
• Jax, Apache-2.029
• matplotlib, Misc30
24https://info.arxiv.org/help/license/index. html
25https://github.com/Tiiiger/bert_score/blob/ master/LICENSE
26https://github.com/huggingface/datasets/blob/ main/LICENSE
27https://github.com/facebookresearch/fairseq/ blob/main/LICENSE
28https://github.com/google/flax/blob/main/ LICENSE
29https://github.com/google/jax/blob/main/ LICENSE
30https://github.com/matplotlib/matplotlib/ blob/main/LICENSE/LICENSE
• NLTK, Apache-2.031
• NumPy, BSD-3-Clause32
• PDFMiner, MIT33
• PubMed, arXiv dataset, Apache-2.034
• Pytorch Lightning, Apache-2.035
• PyTorch, Misc36
• ROUGE Metric, MIT37
• scikit-learn, BSD-3-Clause38
• SciPy, BSD-3-Clause39
• Scrapy, BSD-3-Clause40
• seaborn, BSD-3-Clause41
• Sentence Transformers, Apache-2.042
• spaCy, MIT43
• Stanza, Apache-2.044
• TensorFlow, Apache-2.045
• Transformers, Apache-2.046
31https://github.com/nltk/nltk/blob/develop/ LICENSE.txt
32https://github.com/numpy/numpy/blob/main/ LICENSE.txt
33https://github.com/euske/pdfminer/blob/ master/LICENSE
34https://github.com/armancohan/ long-summarization/blob/master/LICENSE
35https://github.com/Lightning-AI/ pytorch-lightning/blob/master/LICENSE
36https://github.com/pytorch/pytorch/blob/main/ LICENSE
37https://github.com/li-plus/rouge-metric/blob/ master/LICENSE
38https://github.com/scikit-learn/scikit-learn/ blob/main/COPYING
39https://github.com/scipy/scipy/blob/main/ LICENSE.txt
40https://github.com/scrapy/scrapy/blob/master/ LICENSE
41https://github.com/mwaskom/seaborn/blob/ master/LICENSE.md
42https://github.com/UKPLab/ sentence-transformers/blob/master/LICENSE
43https://github.com/explosion/spaCy/blob/ master/LICENSE
44https://github.com/stanfordnlp/stanza/blob/ main/LICENSE
45https://github.com/tensorflow/tensorflow/ blob/master/LICENSE
46https://github.com/huggingface/transformers/ blob/main/LICENSE
• UniEval, MIT47
47https://github.com/maszhongming/UniEval/blob/ main/LICENSE

heading: Future Work
text: Exploring suitable evaluation methods for long document summarization is a future research direction