What Are You Token About? Differentiable Perturbed Top-k Token Selection for Scientific Document Summarization (2024)

Abstract: Scientific document summarization aims to condense complex and long articles in both technical and plain-language terms to facilitate the accessibility and dissemination of scientific findings. Existing datasets suffer from a deficiency in source heterogeneity, as their data predominantly stem from a single common resource, hindering effective model training and generalizability. First, we introduce SCILAY, a novel dataset that includes documents from multiple natural science journals with expert-authored technical and lay summaries. Second, we propose PRUNEPERT, a new transformer-based model that incorporates a differentiable perturbed top-k encoder layer to prune irrelevant tokens in end-to-end learning. Experimental results show that our model achieves a nearly 2x speed-up compared to a state-of-the-art linear transformer, remaining comparable in effectiveness. Additional examinations underscore the importance of employing a training dataset that includes different sources to enhance the generalizability of the models. Code is available at https:// github.com/disi-unibo-nlp/sci-lay.

heading: 1 Introduction
text: Abstractive summarization aims to meticulously condense documents by discerning and rephrasing their salient points. Although this task is relatively easy when summarizing concise texts—such as news facts (Grusky et al., 2018; Narayan et al., 2018)—synthesizing scientific articles presents formidable challenges for humans (Altmami and Menai, 2022). In fact, experts must conduct thorough reviews, encountering technical terms and formulas (Yasunaga et al., 2019), to fully grasp the foundational information contained within the text.
To mitigate the considerable time and effort required for this endeavor, scientific document summarization (SDS) emerges as an indispensable tool.
∗These authors contribute equally to this work.

heading: Summarize
text: SDS refers to the automatic production of technical and plain-language synopses from the scientific literature, including the generation of abstracts (Gharebagh et al., 2020; Frisoni et al., 2023), systematic literature reviews (Moro et al., 2022, 2023e), and journalistic reports (Dangovski et al., 2021). It plays a pivotal role in improving accessibility to the latest research findings, either by assisting experts quickly acquire the desired information or by helping the general public understand complex research topics. For example, in-domain professionals require precise technical summaries laden with specialized jargon. Conversely, nonexperts typically seek more simple syntheses with layman’s terminologies, complemented by contextual explanations that enhance comprehension.
To promote SDS research, several public benchmarks have been proposed over the years. However, such corpora exhibit at least one of the following limitations: (i) they are built to offer either technical (Cohan et al., 2018) or lay sum-
9427
maries (Cardenas et al., 2023); (ii) the documents come from the same source, e.g., ELIFE (Goldsack et al., 2022), limiting model generalizability in production. In response, we introduce SCILAY (Figure 1),1 a new open-access high-quality SDS corpus distinguished by the following characteristics: (i) it comprises author-written and expert-checked summaries of both types that carefully follow journal guidelines; (ii) it covers multiple domains from different sources (e.g., Nature Communications, PLOS Genetics, MDPI Insects, ASM mBio).
To tackle this benchmark, we explore a new model architecture to generate the summaries. Unlike previous solutions that approach this problem with conventional transformer-based pretrained language models (PLMs), we focus on addressing the following challenges. First, scientific papers exhibit an intricate and extensive structure (Kashyap et al., 2023), with summary-worthy information dispersed throughout the long input. Second, processing time and resource demand increase proportionally with the length of the input (Moro et al., 2023d), which presents a bottleneck in real-world applications for long scientific articles (Moro and Ragazzi, 2022, 2023). To this end, we propose PRUNEPERT (Figure 2), an SDS model that extends a PLM architecture with a token-pruning layer. We incorporate a differentiable perturbed top-k mechanism within the encoder stack, aiming to locate a user-defined percentage of summary-worthy input tokens in end-to-end learning. This method allows the model to process fewer input tokens, enhancing the efficiency and interpretability of PLMs.
In summary, our contributions are twofold. First, we establish a new publicly available dataset tailored for both technical and lay SDS. Second, we propose a novel model that learns to select and use only a fraction of tokens for summary generation. Through rigorous evaluations and ablation studies, we demonstrate that our proposed model can generate coherent and informative summaries
1The dataset is available at https://huggingface. co/datasets/disi-unibo-nlp/SciLay.

heading: Technical summary Lay summary
text: comparable to state-of-the-art baseline solutions, registering a 2x speed-up in computational time. Overall, this work contributes to the progression of SDS, offering users more efficient tools for crafting diverse forms of scientific summaries.

heading: 2 SCILAY
text: SCILAY is a new dataset created to summarize scientific papers for both technical and lay audiences.
Collection Unlike existing datasets in the literature (see Table 1 for a comparison), we curate a comprehensive and varied collection of journals from different publishers (see Figure 1). We scrape articles from the PubMed Central repository2—which archives literature from biomedical and life sciences journals—and parse the instances from XML to JSONL format. We then perform a rigorous cleaning phase, discarding samples with missing attributes and outliers based on the length of the document and summaries. Each instance of the dataset includes the full article text, lay and technical summaries, affiliated journal, keywords, Digital Object Identifier (DOI), and a unique identifier in the PubMed Central library database (PMCID).3 We perform an 80-10-10 train/validation/test split by stratifying on the journal type, obtaining 35,026/4380/4384 instances. More details on the dataset are given in Appendix.
Readability We compare the readability of technical with their lay counterparts using the following standard metrics: Flesch-Kincaid Grade Level (FKGL, Kincaid et al., 1975), Coleman-Liau Index
2https://www.ncbi.nlm.nih.gov/pmc/ 3Since not all articles on PubMed Central have lay summaries, our dataset comprises instances that originally include the source article and both a technical and lay synthesis.
(CLI, Coleman and Liau, 1975), Dale-Chall Readability Score (DCRS, Dale and Chall, 1948), and Gunning Fog Index (GF, Gunning, 1952). These assessment metrics gauge the approximate number of years of education required to comprehend a given text. Lower scores signify greater readability; scores falling within the 13–16 range align with the reading proficiency expected at the college level within the US education system. Specifically, (i) FKGL assesses the total count of sentences, words, and syllables contained within the text; (ii) CLI is determined by the number of sentences, words, and characters; (iii) DCRS evaluates readability by considering the average sentence length and the presence of familiar words, using a reference table comprising the 3000 most frequently used English words; and (iv) GF calculates the average sentence length and the proportion of “hard words,” defined as those containing more than two syllables. The results described in Figure 3 indicate the necessity of a college-level education to understand even the lay summary. Consistent with previous work on text simplification (Devaraj et al., 2021; Goldsack et al., 2022; Cardenas et al., 2023), the readability of both summaries is comparable; yet, we observe that lay syntheses are more readable across all metrics.
Characterization To compare the alignment of the technical and lay summaries with the input article, we calculate the extractive fragment coverage

heading: SCILAY Tech Summ Lay Summ
text: and density (Grusky et al., 2018). Technically, an extractive fragment is defined as the set of words shared between two texts. Coverage gauges the proportion of words in the summary that constitute an extractive fragment. Density employs the square of common fragment lengths, ensuring that summaries with longer common fragments receive higher values than those with more numerous but shorter common fragments. Furthermore, we compute the percentage of novel n-grams to assess the amount of information present in the summary that is not explicitly stated in the source document. Table 2 illustrates that lay summaries exhibit lower values for both density and coverage compared to their technical counterpart. In contrast, technical summaries demonstrate lower percentages of novel n-grams. These findings suggest that lay summaries manifest a reduced level of information directly replicated from the source document. In other words, lay summaries exhibit greater abstractiveness, requiring a shift in style from the source and the use of rephrasing strategies.

heading: 3 PRUNEPERT
text: In laying the groundwork for understanding our solution (Section 3.2), we provide a conceptual preliminary with the needed foundations.

heading: 3.1 Preliminary
text: Given the generative nature of the tasks at hand, we employ a transformer-based encoder–decoder model (Vaswani et al., 2017), consisting of an encoder E = {e1, . . . , eL} and a decoder D = {d1, . . . , dL}, each with a stack of L identical layers. E(·) transforms the input sequence of symbols x = (x1, . . . , x|x|) into a series of continuous representations hL = {hL,1, . . . ,hL,|x|}. Then, leveraging hL, D(·) generates the output sequence y = {y1, . . . ,y|y|) in an autoregressive manner.
This
article
reviews
the
role
of
positron
.
Positron
is
Specifically, every ei layer encompasses a selfattention module (SELF-ATT) and a feedforward network module (FFN) through a residual connection and layer normalization (LN):
mi−1 = LN (hi−1 + SELF-ATT (hi−1)) ,
hi = LN (mi−1 + FFN (mi−1)) . (1)
In addition, D(·) uses a cross-attention module (CROSS-ATT) between the SELF-ATT and FFN modules, performing multi-head attention over hL:
ci−1 = LN ( m′i−1 + CROSS-ATT ( m′i−1,hL )) ,
(2) where m′i−1 denotes the output of the SELF-
ATT decoder module after being processed by LN and summed to the residual connection.

heading: 3.2 Perturbed Token Pruning
text: We introduce PRUNEPERT (Figure 4), a novel model that prunes the encoder at token-level granularity using a differentiable perturbed top-k selection module. To achieve our goal of end-to-end model training without the inclusion of supplementary auxiliary losses, we allow the model to pinpoint and leverage only the most significant tokens.
Model Architecture We use the transformerbased architecture to inherit the knowledge acquired by PLMs. We then integrate a scorer network Sϕ and a token selection module T into the model encoder E, where ϕ represents the trainable parameters. Sϕ and T are placed at a certain height of E, between the ew and ew+1 layers, where w is a hyperparameter. Mechanically, we feed the continuous representations hw and
obtain relevance scores r = Sϕ(hw) ∈ R|x| for each token in the input sequence x. Subsequently, we retain the indices of the k most salient tokens i = T (r) ∈ [1, |x|]k, where T is a discrete operator.4 To preserve the original ordering of the hidden states, it is necessary for the i values to be sorted, i.e., i1 < iz < iz+1 < ik. The intuition here is that if we had a large tensor of all token embeddings, we could extract the relevant ones using a single matrix multiplication. To this end, we represent each index iz as the corresponding |x|-dimensional one-hot vector Iz , obtaining I = {I1, . . . , Ik} ∈ {0, 1}|x|×k. Thus, the pruned k hidden states are obtained as ĥw = I⊤hw.
Differentiable Top-k The token selection module T described above is non-differentiable. This implies that during the backpropagation of gradients through the model, no adjustments will be made to the parameters ϕ of the scorer S . To enable differentiation through this operator, we use the perturbed maximum method (Berthet et al., 2020), whose forward pass is defined as follows:
Iσ = E [ argmax
I∈C ⟨I, r1⊤ + σZ⟩
] , (3)
where r1⊤ ∈ R|x|×k are the scores copied k times, σ is an hyperparameter that regulates the influence of the noise, and C is the constraint set defined as follows:
4For example, i1 = z implies that the first selected token is the z-th of the input x.
C ={I ∈ R|x|×k, Ia,b ≥ 0,1⊤I = 1, I1 ≤ 1, ∑
a∈[|x|] aIa,k∗ <
∑
b∈[|x|] bIb,k′ ∀k∗ < k′}.
(4) The condition 1⊤I = 1 ensures that each column sums up to 1, while the last constraint imposes the ordering of the i indices. Empirically, we sample u uniform Gaussian noises to perturb r1⊤. For each perturbed input, we run the top-k algorithm and perform the Monte-Carlo estimation of Equation 3 by averaging their results. According to Berthet et al. (2020), the Jacobian associated with the above forward pass is the following:
J = E [ argmax
I∈C ⟨I, r1⊤ + σZ⟩ZT/σ
] . (5)
Utilizing the top-k operator on each perturbed input leads to the generation of the one-hot matrix I. However, the average of these matrices may deviate significantly from the one-hot pattern, particularly in the initial stages of training when the scoring system is indecisive. This results in obtaining the pruned k hidden states ĥw, forming a weighted average of the original hidden states hw. Accordingly, a beneficial consequence emerges wherein backpropagated gradients consider all tokens. This obviates the need to wait several iterations until the appropriate tokens are consistently sampled. At inference time, we perform hard top-k for efficiency reasons, as there is no need for u perturbed repetitions. This leads to a train–test gap; therefore, as suggested in Cordonnier et al. (2021), we linearly decrease σ to 0 at training time, so that no noise is added and the differentiable top-k operator is numerically identical to hard top-k.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Hardware Environment All runs are tracked with Weights & Biases5 and executed on a workstation with a single Nvidia GeForce RTX3090 GPU of 24 GB of dedicated memory, 64 GB of VRAM, and an Intel® Core™ i9-10900X1080 CPU @ 3.70GHz. The operating system is Ubuntu 20.04.3 LTS. To enhance consistency and portability, our development environment is built on top of a docker container with a NVIDIA image.6
5https://wandb.ai 6nvidia/cuda:11.3.1-devel-ubuntu20.04
Evaluation To perform a comprehensive evaluation of model performance, we address various dimensions. First, we report the F1 scores of syntactic metrics such as ROUGE-{1,2,L} (Lin, 2004), also providing R (Moro et al., 2023b), their variance-aware aggregated score. Second, we use the model-based metric BARTScore (Yuan et al., 2021) for semantic coverage, reporting precision, recall, and F1 values. Third, we report the average readability score across FKGL, GF, CLI, and DCRS.7 We also conduct a thorough human evaluation. Finally, we assess the efficiency of the models by monitoring the training runtime. Additional technical details are given in Appendix.
Baselines Despite the popularity of decoderonly architectures driven by large language models (LLMs), recent findings confirm the superiority of encoder–decoder networks for text summarization (Fu et al., 2023). Therefore, we consider two widely-used encoder–decoder solutions: BART (Lewis et al., 2020), a model characterized by a denoising pretraining objective, and PEGASUS (Zhang et al., 2020), a model featuring a pretraining objective tailored to abstractive summarization. To feed sequences longer than 1024 tokens, we leverage the LSG architecture (Condevaux and Harispe, 2023) for both models, inducing O(n) complexity w.r.t. the input length.

heading: 4.2 Discussion
text: Effectiveness Table 3 presents the results on SCILAY. It is apparent that the inclusion of PRUNEPERT does not degrade the performance of PEGASUS; rather, it enhances summary quality, as evidenced by improved ROUGE and BARTScore metrics in the technical and lay summarization tasks, respectively. PRUNEPERT also contributes to the creation of more readable lay summaries, as indicated by the average readability score. Notably, solutions based on PEGASUS demonstrate superior proficiency in generating lay summaries. This phenomenon may be attributed to the increased difficulty of producing technical summaries, which demand greater length and lexical complexity.
Efficiency In addition to evaluating summarization performance, our focus lies on minimizing computational load during training. Figure 5 illustrates the substantial acceleration achieved by
7As typically done in the literature, we did not evaluate the readability of technical summaries since they are intended for a specialized audience, making such a study less pertinent.
0 0.05 0.1 0.15 0.2
Fw
Bw
Pegasus Pegasus + PrunePert
Lay Summarization
Loading [MathJax]/extensions/MathMenu.js
0 0.05 0.1 0.15 0.2 0.25
Fw
Bw
Pegasus Pegasus + PrunePert
Technical Summarization
Figure 5: Average execution time (seconds) per instance throughout the forward (fw) and backward (bw) passes. PRUNEPERT achieves a speedup of up to 46%.
PRUNEPERT in PEGASUS. Notably, this efficiency enhancement is evident in both the forward and backward passes of the model, resulting in a noteworthy improvement ranging from 42% to 46%. Specifically, the most significant boost in absolute terms is observed during the backward pass, which inherently demands more computational time.
Interpretability To gain a deeper understanding of how the token pruning module works, it is crucial to perform a thorough examination of both the retained and discarded tokens. Therefore, in our analysis, we specifically examine the frequency of stopwords and the following parts-of-speech (POS) tags: adjectives, nouns, verbs, adpositions, and determiners. To achieve this goal, we employ ScispaCy (Neumann et al., 2019), a library equipped with pipelines and models designed for scientific documents. Figure 6 illustrates the results of our
NOUN PUNCT VERB ADJ ADP STOPW DET OTHER 0
10
20
30
40
50
Selected Not-selected

heading: Lay Summarization
text: NOUN PUNCT VERB ADJ ADP STOPW DET OTHER 0
10
20
30
40
Selected Not-selected

heading: Technical Summarization
text: Figure 6: The average rate of POS tags (nouns, punctuation, verbs, adjectives, adpositions, stopwords, determiners, and others) processed by PRUNEPERT in both summarization tasks. We observe a tendency to preserve information stored in nouns, adjectives, and verbs.
analysis, showing the average frequency at which each POS tag and stopword appear in both selected and unselected tokens within the input article. As expected, we observe a tendency to retain elevated information content stored in nouns, adjectives, and verbs. On the contrary, other elements essential for ensuring grammatical correctness, yet offering limited informational value, are excluded. This applies both to lay and technical summarization. However,
the latter is distinguished by a higher retention rate of stopwords, likely aimed at ensuring the more pronounced syntactic overlap described in Table 2.
Furthermore, in Figure 7, we examine the positional distribution of the selected tokens. Remarkably, we observe a clear differentiation between technical and lay summarization. Specifically, in lay summarization, there is a higher frequency of tokens within the first 25% of the input, whereas in technical summarization, the trend is reversed. The distribution of the last 75% tokens appears relatively uniform across both tasks, with no significant distinction. Considering the conventional role of the first section as the preamble in an article, it logically follows that, due to the need for additional contextual information in a lay summary, the selection should be particularly guided by it.
Transfer Learning To deepen our understanding of how SCILAY can contribute to SDS, we use it as a dataset for fine-tuning models, which are subsequently evaluated across benchmarks within the field. Further, our goal is to emphasize the role of source heterogeneity in enhancing the generalizability of models when applied to diverse corpora beyond their training origin. In particular, we experiment with the following two fine-tuning settings:
• PEGASUSall: we train PEGASUS using 3000 instances sourced from SCILAY, ensuring that the distribution of journals within this subset is proportional to the entire dataset.
• PEGASUSplgen: we train PEGASUS using 3000 instances exclusively sourced from the PLOS Genetics journal in SCILAY.

heading: Model R-1 R-2 R-L BaS-F1

heading: PubMed
text: We evaluate both configurations using 1000 instances from the test sets of the SDS datasets PUBMED and ARXIV (Cohan et al., 2018). Table 4 shows that PEGASUSall achieves superior performance in terms of ROUGE and BARTScore metrics in all datasets, underscoring the importance of including training instances from multiple sources.
Human Evaluation To qualitatively analyze the summaries generated by PEGASUS and PEGASUS+PRUNEPERT, we conduct a detailed human evaluation study. We randomly select 30 SCILAY’s test set instances and invite three English-proficient annotators with strong NLP competencies in the biomedical domain. Upon reviewing the articles and their corresponding summaries, each evaluator assigns scores to the generated summaries using a Likert scale ranging from 1 (worst) to 5 (best), on three distinct dimensions: (i) Recall evaluates whether the generated summary encompasses all target contents; (ii) Precision examines whether the generated summary includes only the target contents without extraneous or redundant information; (iii) Faithfulness assesses whether the generated summary maintains factual consistency with the original text. Figure 8 presents the results of the human evaluation study, reaffirming the absence of any notable distinction among solutions.

heading: 5 Related Work
text: Scientific Document Summarization SDS has been a long-standing task, mainly focused on the generation of technical summaries, such as abstracts. Cohan et al. (2018) employ a hierarchical encoder to model the discourse structure and an attentive discourse-aware decoder for summary generation. An et al. (2021) integrate information from the source document and its references, using a graph-based citation model. Recently, there has
been a notable shift in focus within the scientific community, expanding beyond traditional text summarization to include simplification. This shift has given rise to various datasets and methodologies to address this gap. The LaySumm task (Chandrasekaran et al., 2020) is a significant initiative that uses a corpus of 572 articles with authorgenerated lay summaries across various disciplines, including the Materials Science, Archaeology, Hepatology, and Artificial Intelligence journals. Zaman et al. (2020) automatically retrieve simplified and condensed versions of articles available on the Eureka Alert website, resulting in 5204 instances sourced from journals such as PLOS-ONE, Nature Communication, and Scientific Reports. Cachola et al. (2020) propose an innovative approach to extreme summarization, emphasizing core elements while omitting unnecessary methodological details. Guo et al. (2021) collected 6695 pairs of systematic reviews with their corresponding plain-language summaries from the Cochrane Database of Systematic Reviews. In alignment with our work, Goldsack et al. (2022) and Cardenas et al. (2023) provide datasets with varying readability levels. The first presents two datasets from biomedical journals (PLOS and eLife), pairing articles with manuallycrafted lay summaries and abstracts. The second covers diverse domains, including Computer Science, Machine Learning, Physics, and Engineering.
Token Pruning In addition to extract-thenabstract methodologies (Moro et al., 2023c), initial attempts to mitigate the computational burden of transformers—even with linear complexity (Beltagy et al., 2020; Huang et al., 2021)—focus on the removal of non-informative tokens, mainly leveraging attention scores. PoWER-BERT (Goyal et al., 2020) uses a scoring function derived from atten-
tion scores to drop tokens based on their impact on others. Luo et al. (2022) extended this approach to Vision Transformers (ViTs), locating and dropping tokens using attention scores and fusing information from different attention heads. Yang et al. (2022) adaptively determined the quantization precision levels of the tokens (i.e., 0 bit, 4 bit, and 8 bit) based on their importance, as gauged by their attention probabilities. Despite their popularity and promising outcomes, these solutions are grounded in predefined heuristics. Hence, we advocate for an alternative research direction that involves automatically learning the token selection module. In this context, TR-BERT (Ye et al., 2021) formulates token reduction as a multi-step problem addressable with reinforcement learning, introducing an additional loss function to maximize rewards. Conversely, Cordonnier et al. (2021) opt for an end-to-end token selection approach that avoids introducing additional losses, relying on perturbation, which, while differentiable, lacks sparsity. Sander et al. (2023), using p-norm regularization, introduce the first differentiable everywhere and the sparse top-k operator. Nonetheless, it is crucial to note that these pruning-based methods are limited to encoder-only architectures, rendering them unsuitable for direct application to generative tasks.

heading: 6 Conclusion
text: We first introduced SCILAY, a new SDS dataset designed to benchmark models in the creation of technical and lay summaries from heterogeneous sources. Second, we present PRUNEPERT, a new PLM enriched with a token-pruning layer within the encoder stack that allows the model to select only a summary-worthy subset of input tokens for synthesis generation. Quantitative and qualitative analysis attest to the comparable performance between our method and a cutting-edge linear transformer; yet, PRUNEPERT is notably more efficient, with an average speed-up of almost 2x. Further investigation reveals the nature of the selected tokens and the importance of having training sets covering multiple sources to enhance model generalizability.

heading: Limitations
text: Our research is focused on the realm of science. Other domains like finance and law (Moro et al., 2023a) often require text simplification; yet, there is currently a dearth of publicly available technical and lay summaries for documents in these fields.
The effectiveness of our suggested PRUNEPERT model is limited by the requirements established by the specific task at hand. Engaging in the summarization of lengthy documents requires the adoption of models that feature an attention mechanism that exhibits linear scaling w.r.t. the length of the input sequence. Consequently, the performance enhancement achieved through token pruning in the forward pass is constrained compared to solutions employing models with quadratic complexity.
The restrictions of using a single 24 GB of GPU RAM and dealing with long input sequences hinder the use of LLMs, which could potentially have improved overall task performance and the effectiveness of the top-k token selection module.

heading: Ethics Statement
text: While PLMs hold promise for enhancing summarization capabilities across diverse domains, it is crucial to acknowledge their limitations in ensuring the accuracy and fidelity of generated summaries. Therefore, we advocate for a cautious approach, recommending that any output produced by our proposed solution—but it applies to every generative model in the literature—undergoes manual scrutiny by domain experts before utilization for any purpose. This ethical precaution is essential to mitigate the risk of disseminating potentially erroneous or misleading information, particularly within clinical and scientific communities where accuracy and reliability are paramount.

heading: Acknowledgements
text: This research is partially supported by (i) the Complementary National Plan PNC-I.1, “Research initiatives for innovative technologies and pathways in the health and welfare sector” D.D. 931 of 06/06/2022, DARE—DigitAl lifelong pRevEntion initiative, code PNC0000002, CUP B53C22006450001, (ii) the PNRR, M4C2, FAIR— Future Artificial Intelligence Research, Spoke 8 “Pervasive AI,” funded by the European Commission under the NextGeneration EU program. We thank the Maggioli Group8 for granting the Ph.D. scholarship to Luca Ragazzi and partially supporting the Ph.D. scholarship granted to Paolo Italiani.
8https://www.maggioli.com/who-we-are/ company-profile

heading: Appendix
text: License SCILAY will be released under the Creative Commons Attribution 4.0 International (CC BY) license. In fact, the documents in our dataset are open-access manuscripts sourced from various resources, each licensed under the aforementioned license, which enables free and unrestricted usage.
Dataset Our SCILAY dataset has the following sources: Nature Communications (NC), PLOS Genetics (PLGEN), PLOS Pathogens (PLPAT), PLOS Computational Biology (PLCB), PLOS Neglected Tropical Diseases (PLNTD), PLOS Biology (PLB), Biology (B), Communications Biology (CB), Scientific Data (SD), mBio (MBIO), Animals (A), Insects (I), and Cancers (C). Table 5 shows split statistics.
In detail, NC is a multidisciplinary journal dedicated to disseminating top-tier research across a wide array of fields, e.g., biological, health, physical, chemical, and mathematical. PLGEN focuses on studies involving humans and investigations of model organisms, ranging from mice and flies to plants and bacteria. PLPAT publishes groundbreaking research that significantly improves our understanding of pathogen biology or interactions between pathogens and hosts. PLCB features works that advance our understanding of living systems on various scales through the application of computational methods, including molecules and cells, and patient populations and ecosystems. PLNTD publishes research dedicated to the pathology, epidemiology, prevention, treatment, and control of neglected tropical diseases, together with relevant contributions to public health and policy. PLB, BIO, and CB publish significant advances across biological sciences. SD shares advances from all areas of natural sciences, medicine, engineering, and social sciences. mBio reflects the vastness of
the interconnected microbial world, covering symbiosis, pathogenesis, energy acquisition and conversion, climate change, geologic transformations, food and drug production, and even alterations in animal behavior. AN is exclusively dedicated to the field of animals, covering aspects of zoology and veterinary sciences. INS releases articles focusing on the biology, physiology, behavior, and management of arthropods, along with their interactions with human societies, plants, and ecosystem services. CAN covers basic, translational, and clinical studies in all types of tumors.
Training Details We list the hyperparameters used for fine-tuning and inference in Table 6.
Metrics Table 7 lists the hyperparameters of the metrics. BARTScore computes the generation probability p(y|x, θ) of a sequence y conditioned on another sequence x, where θ are the weights of a BART model. Due to this generative approach, the evaluation dimensions vary depending on how y and x are defined. We consider the Recall, Precision and F1 settings. Technically, Recall (h → r, p(r|h, θ)) quantifies how easily a gold reference (r) could be generated by the hypothesis (h). Precision (r → h, p(h|r, θ)) evaluates the likelihood that the answer hypothesis could be constructed based on the gold reference. F1 (h ↔ r) is the harmonic mean of recall and precision. The aggregated score R is formally defined as: R = avg(r1,r2,rL)/1+σ2r , where σ2r is the F1 variance. R penalizes model results with discrepant unigram, bigram, and longest common subsequence overlaps.
Quality Control Table 8 showcases examples of the generated technical and lay summaries, employing PEGASUS and PEGASUS+PRUNEPERT.

heading: Future Work
text: 931 of 06/06/2022, DARE—DigitAl lifelong pRevEntion initiative, code PNC0000002, CUP B53C22006450001, (ii) the PNRR, M4C2, FAIR— Future Artificial Intelligence Research, Spoke 8 “Pervasive AI,” funded by the European Commission under the NextGeneration EU program. We thank the Maggioli Group8 for granting the Ph. D. scholarship to Luca Ragazzi and partially supporting the Ph. D. scholarship granted to Paolo Italiani. 8https://www. maggioli. com/who-we-are/ company-profile