Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Papers
Authors: Chen Tang
Shun Wang
Tomas Goldsack
Chenghua Lin
Venue: EMNLP 2023 Main (2023)
Keywords: Biomedical Text Summarisation
Abstractive Summarisation
Knowledge Aggregation
Citation Graph
TL;DR: We propose a novel dataset and a novel framework to study improving biomedical abstractive summarisation with knowledge aggregation from citation papers

Abstract: Abstracts derived from biomedical literature possess distinct domain-specific characteristics, including specialised writing styles and biomedical terminologies, which necessitate a deep understanding of the related literature. As a result, existing language models struggle to generate technical summaries that are on par with those produced by biomedical experts, given the absence of domain-specific background knowledge. This paper aims to enhance the performance of language models in biomedical abstractive summarisation by aggregating knowledge from external papers cited within the source article. We propose a novel attention-based citation aggregation model that integrates domain-specific knowledge from citation papers, allowing neural networks to generate summaries by leveraging both the paper content and relevant knowledge from citation papers. Furthermore, we construct and release a large-scale biomedical summarisation dataset that serves as a foundation for our research. Extensive experiments demonstrate that our model outperforms state-of-the-art approaches and achieves substantial improvements in abstractive biomedical text summarisation.

Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Papers
Chen Tang', Shun Wang', Tomas Goldsack² and Chenghua Lin 'Department of Computer Science, The University of Surrey, UK
2 Department of Computer Science, The University of Sheffield, UK
3 Department of Computer Science, The University of Manchester, UK chen. tangêsurrey. ac. uk, chenghua. 1 in@manchsterr.. ac. uk (swang209, tgoldsack] Jeshef ff field. ac. uk
Abstract
Scientific Papers
Paper Citation Graph
papers papers
Abstracts derived from biomedical literature possess distinct domain-specinf characteris- ties, including specialised writing styles and biomedical terminologies, which necessitate a deep understanding of the related literature. As result, existing language models strug- gle to generate technical summaries that are on par with those produced by biomedical ex- perts, given the absence of domain-specific background knowledge. This paper aims to enhance the performance language models in biomedical abstractive summarisation by ag- gregating knowledge from external papers cited within the source article. We propose a novel attention-ban based citation aggregation model that integrates domain speciiic knowledge from ci- tation papers, allowing neural networks gen- erate summaries by leveraging both the paper content and relevant knowledge from citation papers. Furthermore, construct and release a large-scale biomedical summarisation dataset that serves as foundation our research. Extensive experiments demonstrate that model outperforms state-of-the-ard approaches and achieves substantial improvements in ab- stractive biomedical text summarisation.
Extract Clediee Abiract
Claiis Abstnen
Chtstee Knewints Agpraptiis
Summariiss
DOD
Lungzeg: Medel
Examples
Reforence Summaries:
.... Correlation analysis was performed to explore potantial associatoos between the deprestion symptoms, ansiety symptoms
Citation Abstract:
... pearson's correlation analycis was porformed explore the correlation betwoonn
agu, donation of hospitalication, time diucharge, buseline ammutory marker -...
Figure 1: The overview of our proposed citation knowl- edge aggregation framework. In this framework, lan- guage models are trained incorporate features from both the content of the main paper and the abstracts of its cited papers. The rationale behind this approach is that the cited papers often share relevant research back- grounds, terminologies and writing styles, which can used as good template for the summary generation.
1 Introduction
handling domain-specific terminologies that are prevalent in biomedical texts. Consequently, a plethora of research studies have been conducted with a primary focus on enhancing language qual- ity by better integrating domain-specific knowl- edge in the biomedicine domain (Sotudeh Ghare- bagh et al., 2020; Tangsali et al., 2022; An et al., 2021; Tang et al., 2023b) However, most prior works have predominantly attempted to incorporate knowledge by leveraging additional annotations within the paper content. These annotations include frequent items (Givchi et al., 2022), named enti- ties (Schulze and Neves, 2016; Peng al., 2021). entity relations (Shang al., 201 I), as well as exter- nal knowledge systems such as biomedical ontolo- gies (Chandu et al., 2017) and external terminology
Biomedical text summarisation plays a pivotal role in facilitating the comprehension of the vast and constantly expanding body of biomedical litera- ture (Xic et al., 2022), which poses significant challenge for clinicians and domain experts who strive remain well-informed in their respective fields. To address this challenge, the generation of high-quality summaries from the extensive corpus biomedical literature holds immense potential in supporting research and advancements within the biomedical domain Youn et al., 2021).
One of the key challenges in biomedical natu- ral language generation (NLG) lies in effectively
"Coreesponding author.

---

searching tools (Gigioli al., 2018). Surprisingly, the inclusion of external knowledge derived from citation papers has been rarely explored in previous biomedical studies. Existing corpora for biomed- ical text summarisation are typically constructed in manner that models solely rely on the source article when generating summary. However, as shown in Figure 1, there exists strong connections among papers in the citation network with shared research backgrounds, terminologies, and abstract styles, which will be a useful source of knowledge for improving biomedical abstractive summarisa- tion but not captured in existing datasets.
tional features derived from abstracts of the citation papers. Within this framework, the base language model can effectively leverage both the features of the main paper and the additional domain-specinn knowledge obtained from cited papers. Conse- quently, this integration leads to enhanced perfor- mance in text summarisation. Extensive exper- iments demonstrate that our model outperforms state-of-the-art baselines in abstractive biomedical text summarisation. We also conducted an in-depth quantitative analysis to verify the performance gain obtained by our attention-based citation knowledge enhancement framework'. Our contributions are summarised as follows:
To address this gap in the existing biomedi- cal summarisation dataset, we construct novel biomedical summarisation dataset utilising open- source biomedical literature corpus provided by the Allen Institute'. During the dataset construction process, we applied rigorous filtering criteria to climinate low-quality samples. Specifically, we dis- carded samples with an insufficient number of cita- tions (less than three distinct citations), as well as unqualified papers whose unique identifiers (UIDs) citation UIDs were inaccessible within the cor- pus. Additionally, we designed heuristic rules to select and transform the unstructured raw data cor- pus into structured dataset JsonL format. The final dataset comprises over 10,000 instances, with each instance having an average of 16 citations.
We construct large-scale biomedical litera- ture dataset, which can be used for enhancing biomedical text summarisation with the ex- tracted external knowledge from cited papers.
We propose a novel framework that can effec- tively leverage citation papers to enhance the performance of large-scale language models on abstractive summarisation of biomedical literature.
We conduct extensive experiments to evaluate the effectiveness of our proposed framework, including comparisons with SOTA models and an in-depth analysis of the performance gain achieved by aggregating different quanti- ties of citations.
To the best of our knowledge, this is the largest biomedical literature dataset² specifically tailored for citation paper-enhanced biomedical text sum- marisation. Furthermore, we provide the corre- sponding methods for collecting the citation net- work, including cited papers and their associations. Facilitated by our biomedical summarisation dataset, we further propose a novel approach to biomedical document summarisation whereby we enhance neural models with external domain- specific knowledge in the form of the abstracts of cited papers. Accordingly, attention-based network (Vaswani al., 2017) that dynamically aggregates features extracted from the citation abstracts with the encoded content features of the main paper. This aggregation is achieved by applying attention mechanisms to the associated abstracts of all cited papers, which provides the subsequent summary decoding process with addi-
2 Related Work
In recent years, variety of large-scale pre-trained models (PLMs), such as BART (Lewis et 2019); TS (Raffel al., 2020); GPT-2 (Radford et al., 2019), have demonstrated remarkable performance improvements across various tasks (Loakman et al., 2023; Zhang et al., 2023; Zhao et al., 2023; Tang et al., 2022b) in the Natural Language Genera- tion (NLG) Domain. These PLMs have also been widely applied to biomedical text summarisation. These models, e.g. BioBERT (Lee et al., 2020) and BioBART (Yuan et al., 2022), have achieved remarkable performance by training on extensive biomedical literature corpora, such as Pubmed and MIMIC-II'. However, certain high-level knowl- edge, e.g., the understanding of medical termi- nologies, cannot be adequately captured solely
introduce an
'https://allemai. hatpaai...r//ttaa/r/rrreeee
"Our code and data resources is accessible https:// gi thub. cww/ wtw.//oootteeeeeee twwggg55/biiomdd-aarrrr 'https://pubmed ncbi. nlm. nov. Phttps://physione or wttp:/tooocctttt/miiiiiiiiiiiiii
'The sole viable dataset we have identified is The Text Analysis Conference (TAC) 2014 Biomedical Summarization track (Cohan et al., 2014) comprising mere 313 instances.

---

through the implicit modeling of word probabili- ties. To address this limitation, the improvement of biomedical background knowledge understanding is able to necessitate the integration of additional knowledge systems, such as conceptual ontologies.
Algorithm 1: Construction of BioCiteDB
Input: Samples c, E C: Citation limit ft
Output: Json objects d, E
Initialise D
foreach & in do
Initialise object d. with c;
These ontologies explicitly model representations of domain-specifie knowledge learned by neural networks. Recent studies have proposed incorpo- rating biomedical knowledge, including terminolo- gies (Tang al., 2023b)) and concepts (Chandu al., 2017), to enhance the performance of these language models and bridge the gap between lan- guage understanding and specialized biomedical knowledge. Indeed, several notable works have focused on enhancing summarisation through cita- tions in the open domain, such as An et al. (2021) and Yasunaga et al. (2019). However, it is impor- tant to highlight that the progress of language mod- els in the biomedical domain has been hindered by the limited availability of datasets and resources. This scarcity has impeded the further advancement and improvement of pre-trained language models (PLMs) specifically tailored for biomedical appli- cations. In this study, we require a dataset that contains retrievable citation papers, making tra- ditional raw data corpora such Pubmed and MIMIC-III inadequate. To date, the sole public dataset we could find is the Text Analysis Con- ference (TAC) 2014 Biomedical Summarization track (Cohan et al., 2014). However, this dataset is limited in size comprising merely 313 instances, and is somewhat outdated. Therefore, we con- struct a novel dataset for investigating biomedical citation-enhanced summarisation.
retrieve files fj queue
Initialise object p.
foreach fi in do
if missing elements then break
end
extract distinct citations ri fa
if [r >= R then
Tu.add + pe.citations
extend d, with p:
14
break
15
end
end
D append d: end
foreach
do
foreach 7n in d, citaatioo do
if rv $ D then
12
exclude in
23
end
end
end
Datasets
Train
Val
Test
# Samples
9144
1143
1143
Pupers
4762
Avg. Distinet Citations
6:23
Avg. Total Citations of Doc
16.56
16.85
16.96
Avg. Chunks
Avg. Sentences in Doc
$29.94
526.8
529.1
Ang.
in
9901.96
Avg. Sentences Summary
13.98
Avg. Words in Summary
220.25
222.04
217.28
Table 1:
Data statistics of Biomed Ref dataset. Doc is abbr. of Document, Intro the abbr. intro- duction. Chunks are split by section and subsections in paper.
3 Dataset Construction
3.1 Construction Process
the "Introduction" section, it is considered the most crucial part for generating abstracts; (2) The papers must have least three distinct citations ensure the quality of curated data; (3) The essential elements of the papers, including UID (Pubmed id), Title, Abstract, Sections, and Citations, must be accessible within the raw corpus. As result of this construction process, the dataset D comprises structured data in JsonI. 7 format, with each sample representing an individual paper.
In order to create a dataset containing biomedical literature and its associated citations, we process semi-structured raw corpus released by Allen Institute. We refer to this dataset BioCiteDB throughout the paper. The construction process of the dataset is outlined in algorithm 1, where C represents the raw corpus, and D represents the pro- cessed dataset. To ensure the quality and relevance of the data, the selected papers have to meet the fol- lowing requirements: (1) The papers must include
3.2 Data Statisties
The statistical analysis of our processed dataset is presented in Table 1. Additionally, the distribution
"We select the latest version of CORD-19 Historical Re- leases (2022-06-02 18.7 GB), which can be accessed at https: //ai2-semanti cttmmmnnnicccooaarr.m.. cwcholac---ceeccee wwwsschhooa--rr----- cord-19 19. s3-us-west 2. amazonaws. wwtt://ttto./acccccccccooootctct releases. html.
Phttps://mante fold. how..n/ee/////////////////////// htm.

---

Propective at Soper
Proprttion Paper
Ube Proportion Pee Peper
The Propuction Data Size
Pepurtin Dhit Iestvet
Projection taato to Sraanven
Peoporban Dutta
44.2%
Citationa
Cituiieu Popet
12 Citalious
others
(b)
Figure 2: visulisation of the distribution of citations per paper and the data size associated with each split. To avoid clutter and maintain clarity, we have excluded papers with over 50 citations from the visualization, as they constitute a relatively small proportion the dataset and fall into the long tail category. Some papers in our corpus are only used citations other than the papers in data splits,
categorise them as "others".
citations per paper is visualized in Figure (a), (b), and (c), while the proportions of data size are depicted in Figure 2 (d) of the same figure. The results obtained from both the statistical analysis and visual representations in Table I and Figure both validate the data quality of the constructed dataset, thus indicating the effectiveness of our data construction process and the consistency of the dataset splits. This validation supports the notion that training and inference tasks conducted on this dataset can be regarded as fair and reliable.
tion graph, we propose an algorithm outlined algorithm 2. hopmas defines the maximum num- ber of hops between papers that the algorithm can traverse, and N specifies the maximum number of retrieved papers at each hop. As output, P repre- sents papers as nodes, while citation relationships are represented edges in the network. Due to the high computational cost of processing long doc- uments for summarisation, we set hopmaz to and neighbor to 12, taking into account the limitations of our available computing resources. However, it is worth noting that the attention-based citation aggregation module can be extended to in- corporate Graph Attention Networks (Zhou et al., 2020), which have the capability to integrate multi- layer citation graphs (Zhang et al., 2023).
Algorithm 2: Extracting Citation Graph G.
Input: di E D: hopmes: N
Output: The set related papers
Initialise current hop hop, = 0; double-ended
queue DQ +- (di. hop.)): a queue recording visited nodes VQ: while S do pop wid and hop-. from DQ
Methodology
hopa. hopmaz then
return P end
As illustrated in Figure 3, our proposed frame- work is designed to enhance the performance of the base language model by leveraging the collec- tive knowledge from set of citation papers. For experiments, we select BART (Lewis et al., 2019). widely-used summarization model that has demonstrated promising results in the biomed- ical domain (Goldsack et al., 2022, 2023), as the base model. In this study, we adopt a strategy where we concatenate the abstracts of the citation papers with the input document to form the model's input. This approach is motivated by the goal of en- abling the model to capture and emulate the writing style present in relevant papers. By incorporating this additional information, we aim to improve the model's ability to generate high-quality summaries that align with the conventions and patterns ob- served in the domain-specific literature.
P + (wid, hopa) if P > N then
return P
10
end
I1
get d, by uid
12
VQ - wid
13
foreach 7n E dy-citations do
14
if wid é og and r. E D then
15
P t wid and VQ +- wid end
16
end
is
3.3 Extract Citation Graph
Scientific papers are intricately connected through citation relationships, forming network of inter- connected nodes. This citation graph provides valu- able insights into the relatedness of papers. In order to retrieve relevant papers within this cita-

---

LM
doc;
Citation Abstracts
Add Nara
LM mnc.
Add Nerm
doe;
abs"
Malii Head Attaniin
Tum
L.M
Loyer
Head
Add & Norm
fratures mensure
doc,
Atteation atta) for each ab.;
Apply acen) to aggregate fentures
Summaries
Figure 3:
The illustration of our proposed framework. doc is the abbr. of the input document referring to the paper content. abs is the abbr. of the abstracts and abs, denotes the abstracts of citation papers.
4.1 Task Definition
Encoding In order to capture the relevance of each cited abstract, we employ an attention mecha- nism to measure the importance of d; with respect to absg. The attention score is denoted attn),, and the process of aggregating knowledge illus- trated as follows:
The task formulated as follows: Given a paper document d, E as the input, where repre- sents the paper corpus, and di denotes the i-th paper. In addition, the citations papers D {di, dg...., d)) are also provided as the input. The abstracts of di D denoted absī- Either dj d consists of sequence of words repre- sented as X {z1, 22, .T) where 2t denotes 1-th word X. The goal is to generate a summary = {y1, .... y by modeling the conditional probability distribution X e d,, X E D
Eq = concat [Eq.. Equ()
(4)
Q LMen (EQ), Q E NxLMM
(5)
where Eq denotes the matrix of embeddings for all composed Qj. and it is encoded by the language model encoder to generate the encoded features Q.
4.2 Knowledge Aggregation from Citations
QCLS First Pirst_Pool((Q).. QLSS N
(6)
Input At the initial stage, both the input docu- ment d; and its retrieved N citation abstracts abs are concatenated and encoded by language models. Byte-Pair Encoding (Radford al., 2019) is imple- mented in the transformation from text into fixed word embeddings:
Alln = QCL.S Alta E ERN
(7)
Alln = softmax(AUln), Alln e Nx1 F =
(8)
(9)
In the above equations, First_Pood collects fea- (1) tures that represent the global context of the input d, and absf) pairs. As the hidden states of the neu- (2) ral encoder, Q incorporates features from both the documents and the abstracts. Therefore, the rep- resentations of the first position in Q (represented as QCI.S) correspond to the global context token T TokCLS, The attention logits matrix is obtained by applying trainable parameter / E Mx1 to the features of QCLS, After applying the soflmax function for normalization, Attn represents the importance of the input features and is used reweight the original encoded features Q, resulting in the final features F.
Edoc = LMemb CLS xx E d:))
Eabez LMen ( oTo ABS,
ABS Iz E ubs)])
Eq, concat (Edeet(Edocr Eabs;)
where L.eenb represents the module responsi- ble for tokenising and converting words into sub- word embeddings. CLS is special token that signifies the global context tag in the input text. ABS special token used indicate the sep- aration between the input document and the cited abstracts. EQ, denotes the embeddings generated for the j-th (j E [1, NJ) document abstract pair.

---

4.3 Summary Generation
be considered as the lower bound of the perfor- mance. ORACLE select sentences from the in- put document and compose a summary with the highest score", which is the upper bound of ex- tractive summarisation systems. The PLM models serve as baselines for abstractive biomedical sum- (10) marisation. The Long-Document Transformer (LED) is Transformer-basee models which are (11) able process long sequences due their self-
In line with other abstractive summarization sys- tems, employ an auto-regressive decoder to generate summary tokens ys sequential manner. The process is described follows:
H. = Decoder(yet, F)
P(mnlve X) softmax(H,WW) HWWD)
sampling P(u: Vct. F)
(12) attention operation (Beltagy al., 2020). PE- GASUS (Zhang al., 2020) pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective, which is tailored for abstractive text sum- marisation. BART (Lewis et al., 2019) is widely used PLM model based on denoising autoencoder that has proved effective for long text generation tasks. Pubmed-X refers to several PLM-based baselines that have been pre-trained on large- scale biomedical literature corpus Pubmed (Cohan et al., 2018) (the size of the dataset is 215k), where X denotes the name of a PLM. Additionally, we include ChatGPT for comparison. However, as it is close-source source and very expensive for training, were unable to use ChatGPT the base lan- guage model to fine-tune on our dataset. Instead, compare the outputs of ChatGPT in zero-shot setting.
y/8
where represents the current time step. corre- sponds to the input, consisting of the words from dj and absq, .... absş, provided to the neural model. H, refers to the hidden state of the decoder module at time step t. This state is computed by the lan- guage models using the infused features F, which encapsulate the information from the input docu- ment and its cited abstracts, along with the previ- ously predicted tokens yes. W denotes a train- able parameter, and P(yllee,, F) represents the probability distribution over the vocabulary, which includes special tokens. Employing a sampling strategy, such as argmaz, we obtain the predicted token Vr-
4.4 Training and Inference
Finally, as shown Figure the neural model trained to fit on the citation-enhaneed training set by the following objective function:
Evaluation Metrics In the domain of text sum- marisation (Sun et al., 2021; Tang et al., 2022a; Xic et al., 2022), ROUGE (Lin, 2004) is the most used metric for the evaluation of generated summaries. For evaluating the quality of the generated sum- maries, we implement the ROUGE metrie with the python package of rouge_score. Specifically, we report the unigram and bigram overlaps (ROUGE-1 and ROUGE-2, respectively) between the gener- ated summaries and the reference (golden) sum- maries. Additionally, we include the longest com- mon subsequence (ROUGE-L) metric to evaluate the fluency of the generated summaries. For each ROUGE metric, we provide fine-grained measure- ments of precision, recall, and F-values, offering comprehensive assessment of the summarisation performance.
L
log P(nlBer. X)
(13)
N
where C is the cross-entropy loss employed to train the model in modeling the conditional probabilities over the token sequence P(yilyee, F). By min- imizing L, the language model learns predict the referenced abstract corresponding the input document.
5 Experiment
5.1 Experimental Setup
Baselines We include a range of competitive PLM models baselines. We also provide the results of two rule-based systems, namely LEAD-3 and ORACLE, which serve as benchmarks rep- resenting the upper and lower bounds of model performance. LEAD-3 extracts the first 3 sen- tences from the input as the summary, which can
In addition to the ROUGE metric, we conduct an extensive automatic evaluation utilising a broader range of evaluation metrics. Specifically, we em-
*In this study, the score referenced by ORACLE is cal- culated as the mean value of the ROUGE-1, ROUGE-2, and ROUGE-L scores.

---

ROUGE-11
ROUGE-21
ROUGE-LA
Models
PPL.
Precision
Recall
F-value
Precision
Recall
F-value
Precision
Recall
F-value
LEAD-3
0.5512
0.1838
0.2645
0.2039
0.0647
0.0941
0.4979
0.1646
0.2374
ORACLE
0.5669
0,4121
0.4676
0.2478
0.1764
0.2015
0.5195
0.3762
0.4276
ChatGPT
0.4965
0.3899
0.4242
0.1720
0.1358
0.1471
0.4551
0.3564
0,3880
LED
11.36
0.3250
0,3129
0.3150
0.0940
0.0909
0.0912
0,2934
0.2822
0.2844
PEGASUS
12.54
0.3063
0.3036
0.3003
0.0837
0.0833
0.0821
0.2715
0.2681
0.2657
BART
10.89
0.3581
0,2963
0.3171
0.0862
0.0709
0.0760
0.3276
0.2719
0.2907
Pubmed-LED
11.01
0.3462
0.3394
0.3395
0.0877
0.0860
0.0859
0,3109
0.3044
0,3047
Sabmed-PEGASUEE
18.27
0.3806
0,2463
0.2426
0.0980
0.0635
0.0752
0.3438
0.2188
0.2618
Pubmed- Tubmed-BART
10.80
0.3789
0,3242
0.3426
0.1027
0.0875
0.0926
0,3464
0.2963
0.3134
Puhmed-BARE
.W one citation
11.51
0.3816
0.3288
0.3461
0.1070
0.0926
0.0971
0.3473
0.2998
0.3154
citation agg.
10.54
0.3758
0.3427
0.3522
0.1039
0.0946
0.09731
0.3450
0.3143'
0.3233
Table Automatic evaluation based on ROUGE scores. LEAD-3, ORACLE, and ChatGPT were excluded from the performance comparisons as they were not trained on the datasets. However, we use them as reference models to provide insights into the potential performance achievable on our datasets. For each metric, the best overall score is highlighted in bold, and the baseline score is underlined. / indicates the higher/lower the better, respectively. one citation to denote the input configuration where the document is composed with one randomly selected citation abstract. citation agg. denotes our proposed citation abstract aggregation framework. denotes that the citation-enbanced model results are statistically significant with respect to the base model (Pubmed-BARe -BARe by way of Mann- WhinnWy U test.
play BERTScore (Zhang al., 2019) (BeS) and BartScore (Yuan et al., 2021) (BaS) to assess the quality of the generated outputs. We also intro- duce some readability metrics, e.g. Plesch-Kincaid (FLK) and Coleman Index (CLI), to evaluate the readability of the generated text. This com- prehensive evaluation allows for more robust as- sessment of the summarisation performance across multiple dimensions.
Referenced BeS
Unreferenced FLK]
Model
BaSt
CLI
Golden
100
-0.27
16.49
15.69
ChatGPT
85.56
-3.11
16.19
16.31
Pubmed-LED
82.85
-3.37
16.33
15.31
PubAec-PEGGASUS
81.97
-3.42
15.78
14.52
Pubmed-BARK
83.34
-3.36
14.45
13.32
Pubmed-BART
citation
83.34
-3.36
14.78
13.51
citation agg.
83,60
-3.35
13.82
13,20
Table 3: Automatic evaluation
metrics. BeS and BaS denote the Fl values of BERTScore and BartScore, respectively. FLKK and CLI denote the read- ability scores of Flesch Kincaid and Coleman- Liau In- dex, respectively.
5.2 Implementation Details
All of the pre-trained models used are restored from the publicly available checkpoints Hug- ging Face". The checkpoints we selected include: LED PEGASUS"I B2 Pubmed-LED PEGASUS and Pubmed-BART To make the comparison fair, all input text is chunked according to the minimal input size lim- itation of selected language models. In our ex- periments, it is BART (1024 tokens). Models are trained for up to 10 epochs on Tesla A40 ma- chine, which has 40 GB GPU memory, and the best checkpoints are kept based on the perplexity of generated responses during validation for the
generation on the testset. The batch size 16, and the learning rate is le-4, with the Adam optimizer selected for training. For more details, please refer to the Appendix A.1.
5.3 Automatic Evaluation
The results of all experiments are presented Table 2. can be observed that our proposed framework (-w citation agg.) significantly outper- forms all baseline models across all ROUGE scores (FI scores), indicating substantial enhancements in the summarisation capability of biomedical papers. To be more specific, the incorporation of citation knowledge has contributed to a substantial improve- ment recall, with ROUGE-1 exhibiting 5.7% increase and ROUGE-2 demonstrating an 8.1% in- crease. This suggests that the integration of citation
"https://huggiiggaass http:///ggnnaaacc..ccoooeeeele co/models
"https://huggingte" ctt.lleaaaaaaaaaaa//aaaaaaaaaaaa
led led-base-16684
1I "nttps://hugginggfaee
httpee/ee/ooooooggssssssss//// x-base
#nttps://huggingh
sumpubmed__
"https://hugging co/nse30/
port-base-fineturee pubmed

---

knowledge has facilitated the utilisation of more similar expressions extracted from the reference abstracts.
Human Evaluation
Score (1 5)
Flu
Rea
Rel
Inf
Golden ChatGPT
4.86"
4.70"
5.0"
5.0"
3.77
4.33"
4.33"
In addition, within framework, the lan- guage model achieves substantially lower perplex- ity (PPL) and ROUGE-L scores, signifying an im- provement in the language quality and reduced con- fusion during summary generation. We hypothesise that the decrease PPL and ROUGE-L indicates that the language model has learned writing styles and relevant biomedical terminologies by referring to the abstracts of cited papers.
Pubmed-LED
2,70
2.67
3.83*
3.70
Pubmed-PEGASIU
2,33
2.33
3.53"
3.27
Pubmed-BARY
2.73
2.73
3.77"
3.57
"ubmed-BARK
.W one citation
2.87
2.77
3.93"
3.63
.w citation agg.
2.96
2.93
4.0"
3.83"
Table 4: Results of Human Evaluation. Flu, Rea, Rel and Inf denote Fluency, Readability, Relevance, and Informativeness, respectively, The best scores in bold, and the second best are underlined. ChatGPT and Golden are not included in comparison. We cal- culate Fleiss' Kappa 6 for each metric. The majority of results demonstrate a moderate level of agreement (k E (0.4, 0.6)), and results with higher level of agree- ment are marked with
Regarding the ablation study, -w one citation yields slight improvement compared to the base- line model Pubmed-BART but exhibits a higher perplexity. This observation suggests that the direct inclusion of random citation content may introduce certain noise. In contrast, attention-base -based mech- anism enables the neural networks to dynamically select and aggregate important information from multiple citations, effectively addressing confusion issues associated with additional inputs.
human evaluation. This evaluation focuses on four key aspects: fluency, readability, relevance, and informativeness. Fluency assessment aims to mea- sure the overall quality of the language used in the summaries. Readability evaluation determines the extent which the summaries are easily un- derstandable by readers. Relevance assessment examines whether the content of the summaries is pertinent and aligned with the content of the input document. Informativeness measurement evaluates the extent to which the generated summaries pro- vide sufficient and meaningful information derived from the given input. By incorporating human eval- uation, we assess subjective aspects of sum- mary quality that automated metrics may not fully capture.
In Table 3, we present the results of additional evaluation metrics. BertScore and BartScore, as machine learning-based metrics, measure the se- mantic similarity between the generated summaries and the reference abstracts. Flesch-Kincaid and Coleman-Liau metrics assess text readability on a vocabulary level. Across all these metrics, -w citation agg. outperforms all bascline models, showcasing the advantages of introducing citation knowledge with our framework. Further analysis within the "Pubmed-BAAR model reveals that us- ing a single citation results slight decrease in BERTScore and BartScore, along with a slightly higher Flesch-Kincaid score (14.78) and Coleman- Liau Index score (13.51). However, employing citation aggregation leads to improvements across all metrics, with BERTScore (83.60), BartScore (- 3.35), Resch-Kincaid score (13.82), and Coleman- Liau Index score (13.20). This analysis confirms our initial hypothesis, that directly introducing a random citation may introduce noise that hampers model performance, while our aggregation model comprehensively considers all citation papers, ef- fectively reducing the random noise introduced by single citation.
Considering the difficulty of evaluating gener- ated summaries, which requires thorough under- standing of the content in both the source papers and the summaries, it is imperative that human eval- uators possess a strong background in academic writing and biomedical knowledge. We invite 3 qualified evaluators by snowball sampling to rate 30 randomly sampled instances from the testset. In order to minimise biases and increase inter- annotator agreement, the evaluators were provided with the same annotation guide (see Appendix A.3). The results of the human evaluation are presented in Table 4. It can be observed that both the w one citation and citation agg. models exhibit superior performance compared to other baseline models, thereby affirming the effectiveness of our proposed framework.
5.4 Human Evaluation
order to obtain a more comprehensive evaluation of the generated summaries, we also incorporate

---

Iuproccunnt LEROUGES
Datta BOUDEE
Mor Citatheess
The of Unattess
The
The
(c)
Figure 4: (a), (b) and (c) illustrate the bar charts depicting the performance enhancement achieved by the citation agg. method over the PubmedBART model. The improvement is calculated as the difference between the scores obtained by -w citation agg. and PubmedBART. In these bar charts, we report ROUGE scores. Additionally, Figure I(d) exhibits the smoothed curve of the ROUGE scores data, obtained using Gaussian kernel smoothing technique. Due to the limit of GPU memory, we limit the input of the citation aggregation network to 12 citation papers.
To delve further into the ovaluation, the metrics of Relevance and Informativeness underscore the improved capability to extract relevant information from the input content and generate comprehensive abstracts. Additionally, the fluency and readabil- ity metrics assess the language quality, indicating that the language model generates abstracts that are more coherent and natural. However, it is im- portant to note that the tested Pretrained Language Models (PLMs) exhibited a notable disparity in lan- guage quality when compared to the performance of ChatGPT. This discrepancy the substantial difference in model size, with Chat- GPT having 130 billion parameters, whereas the tested PLMs have less than billion parameters.
Conclusion
In conclusion, we proposed a novel attention- based citation aggregation model that incorporates domain-specific knowledge from citation papers. By integrating this additional information, our model enables neural networks to generate sum- maries that benefit from both the paper content and the associated knowledge extracted from citation papers. Furthermore, introduced specialized biomedical summarisation dataset, which served valuable resource for evaluating and advancing our research. The effectiveness our approach was demonstrated through extensive experiments, where our model consistently outperformed state- of-the-an art methods in biomedical text summarisa- tion. The results highlight the significant improve- ments achieved by leveraging knowledge from ci- tation papers and the potential for our model enhance the understanding of biomedical literature through natural language generation techniques.
be attributed to
5.5 In-depth Analysis
To further investigate the impact of the citation knowledge aggregation module, we conduct an evaluation to assess the improvement in the gener- ated abstracts. This evaluation involves comparing the performance of proposed framework, de- noted as -w citation agg., against the base model Pubmed-BART using ROUGE scores. The results, presented in Figure as (a), (b), and (c), illustrate the increase in ROUGE scores (F value) for differ- ent numbers of citations. The inclusion of citations is shown to have positive effect on the abstract generation process. The Gaussian kemnl smoothed increasing curve, depicted in Figure 4 (d), indi- cates clear trend: as more citation abstracts introduced, the language model exhibits greater improvements. The results highlight the potential leveraging citation information to enhance the quality generated abstracts.
A Acknowledgements
Chen Tang supported by the China Scholar- ship Council (CSC) for his doctoral study (File No. N0.220200110033) We also gratefully acknowl- edge the anonymous reviewers for their insightful comments.
Limitations
In the field of text summarisation, two main ap- proaches are commonly employed: extractive sum- marisation and abstractive summarisation. While extractive summarisation composes summaries by directly selecting sentences from the input content, abstractive summarisation generates summaries

---

that are not bound to the input content, providing greater flexibility but posing challenges in manage- ment and control. In this work, due to resource and time constraints, we focused on implementing an abstractive summarisation model and did not further conduct experiments to develop an extrac- tive summarisation counterpart using our proposed algorithm. However, it is worth noting that our proposed approach has shown promising results, emphasizing the importance of leveraging citation papers enhance the performance of language models in generating high-quality biomedical sum- maries. Theoretically, the aggregation of knowl- edge from citation papers can also be beneficial for extractive summarization approaches.
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pages 615 leans, Louisiana. Association for Computational Lin- guistics.
615-621, 1,1, New Or-
Arman Cohan, Luca Soldaini, and Nazli Goharian. 2014. Towards citation- baaoon summarization of biomedi- cal literature. In Proceedings of the Text Analysis Conference (TAC"14). 14).
Jay De Young. Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. 2021. MS^2: Multi- document summarization of medical studies. In Pro- ceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pages 7494- 7513, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Paul Gigioli, Nikhita Sagar, Anand Rao, and Joseph Voyles. 2018. Domain- aware abstractive text sum- marization for medical documents. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 2338-2343. IEEE.
Ethies Statement
Our new dataset is derived from existing pub- licly available corpus released by the Allen Insti- tute, which is a comprehensive biomedical liter- ature corpus licensed under the Apache License
Azadeh Givehi, Reza Ramezani, and Ahmad Baraani- Dastjerdi. 2022. Graph-based abstractive biomedical text summarization. Journal of Biomedical Informat- ics, 132:104099.
2.0. We have diligently adhered to the terms and conditions of the license and followed all provided instructions. Furthermore, we have familiarized ourselves with and acknowledged the ACM Code of Ethics and Professional C We approach our professional responsibilities with utmost seri- ousness, ensuring that our study upholds ethical principles in every aspect.
Tomas Goldsack, Zheheng Luo, Qianqian Xie, Car- olina Scarton, Matthew Shardlow, Sophia Ananiadou, and Chenghua Lin. 2023. Overview of the biolay- summ 2023 shared task on lay summarization of biomedical research articles. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 468-477.
Tomas Goldsack. Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Cor- pora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 10589-10604, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
References
Chenxin An, Ming Zhong, Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. 2021. Enhancing scientific papers summarization with citation graph. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 12498-12506
Henglin Huang. Chen Tang, Tyler Loakman, Frank Guerin, and Chenghua Lin. 2022. Improving Chi- nese story generation via awareness of syntactic de- pendencies and semantics. In Proceedings of the 2nd Conference of the Asia-Pacijt Chapter of the A Asociation for Computational Linguistics and the 12th International Joint Conference on Natural Lan- guage Processing ( (Volume 2: Short Pupers), pages 178-185, Online only. Association for Computational Linguistics.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The doc-meen transformer. arXiv preprint arXhv:2004.05/55
Khyathi Chandu, Aakanksha Naik, Aditya Chan- drasekar, Zi Yang. Niloy Gupta, and Eric Nyberg. 2017. Tackling biomedical text summarization: OAQA at BioASQ SB. In BioNI. 2017, pages 58- 66, Vancouver, Canada, Association for Computa- tional Linguistics.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 3634):
Arman Cohan, Franck Dernnncour, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy.
"nttps: http://wwwwwcc/// acm. wwg.aa//cooooooo..........oo wttgg//ooooooooooooee -t-hcc

---

Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence- tore--to--oooee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13:: 13461.
Chen Tang, Frank Guerin, Yucheng Li, and Chenghua Lin. 2022a, Recent advances in neural text gen- eration: A task-agnostic survey. arXiv preprint arXiv:2203.0300
Chin-Yew Lin. 2004, ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chen Tang. Chenghua Lin, Henglin Huang, Frank Guerin, and Zhihao Zhang. 2022b. EtriCA: Event- triggered context-aware story generation augmented by cross attention. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 5504 5518, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Tyler Loakman, Chen Tang, and Chenghua Lin. 2023. TwistList: Resources and baselines for tongue twister generation. In Proceedings of the 6l Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 579-589, Toronto, Canada. Association for Computational Linguistics.
Chen Tang, Hongbo Zhang, Tyler Loakman, Chenghua Lin, and Frank Guerin. 2023a. Enhancing dialogue generation via dynamic graph knowledge aggrega- tion. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Vol- ume Long Papers), pages Canada. Association for Computational Linguistics.
Keqin Peng, Chuantao Yin, Wenge Rong, Chenghua Lin,
Toronto,
Deyu Zhou, and Zhang Xiong. 2021. Named entity aware transfer learning for biomedical factoid ques- tion answering. IEEE/ACM Transactions Com- putational Biology and Bioinformatics, 19(4)23365- 2376.
Chen Tang, Hongbo Zhang, Tyler Loakman, Chenghua Lin, and Frank Guerin. 2023b. Terminology-awar medical dialogue generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):
Chen Tang, Zhihao Zhang. Tyler Loakman, Chenghua Lin, and Frank Guerin. 2022c. NGEP: A graph- based event planning framework for story generation. In Proceedings of the 2nd Conference of the Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), pages 186-193, Online only. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67,
Frederik Schulze and Mariana Neves. 2016, Entity- supported summarization of biomedical abstracts. In Proceedings of the Fifth Workshop on Building and Evaluating Resources for Biomedical Text Min- ing (Bi01xMM20166, pages 40-49, Osaka, Japan. The COLING 2016 Organizing Committee.
Rahul Tangsali, Aditya Jagdish Vyawahare, Aditya Vyankatesh Mandke, Onkar Rupesh Litake, and Dipali Dattatray Kadam. 2022. Abstrac- tive approaches to multidocument summarization of medical literature reviews. In Proceedings of the Third Workshop Scholarly Document Processing. pages 199-203, Gyeongju, Republic of Korea. Association for Computational Linguistics.
Yue Shang. Yanpeng Li, Hongfei Lin, and Zhihao Yang. 2011. Enhancing biomedical text summariza- tion using semantic relation extraction. PLoS one, 6(3)x223862.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Sajad Sotudeh Gharebagh, Nazli Goharian, and Ross Filice. 2020. Attend to medical ontologies: Content selection for clinical abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1899- 1905, Online. Association for Computational Linguis- tics.
Qianqian Xie, Jennifer Amy Bishop, Prayag Tiwari, and Sophia Ananiadou. 2022. Pre-trained language models with domain knowledge for biomedical ex- tractive summarization. Knowledge-Bord Based Systems. 252: 109460.
Edward Sun. Yufang Hou, Dakuo Wang. Yunfeng Zhang, and Nancy X. R. Wang. 2021. D2S: socumenn-to-slidde generation via query- query-based text summarization. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 1405-1418, Online. As- sociation for Computational Linguistics.
Bohao Yang. Chen Tang, and Chenghua Lin. 2023a. Improving medical dialogue generation with ab- stract meaning representations. arXiv preprint arXiv:2309.1006
Bohao Yang. Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. 2023b. Effective distillation

---

of table-based reasoning ability from llms. arXiv preprint 1rXih:2309.33185
with the python package rouge_ The readability scores such as Flesch-Kincale (FLK) and Coleman-Liau Index (CLI), are implemented with the python package py-readability-mere metrics.. BertScore bert_score, and BartScore is from the GitHub repository of https: //gi neulab/BARTScom
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexan- der R Fabbri, Irene Li, Dan Friedman, and Dragomir Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for sci- entific paper summarization with citation networks. Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7386-7393.
http://ggghubb.oo..... thub. com/
A.2 Automatic Evaluation
Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022. Biobart: pretraining evaluation a biomedical generative language model. arxiv preprint arXiv:2204.03300.
Table 6 shows the full results of BertScore and BartScore.
A.3 Human Evaluation
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text text gener- ation. Advances in Neural Information Processing Systems, 34:27263-27277.
In addition to automatic evaluation metrics, we conducted comprehensive human evaluation assess the quality of biomedical summarization generated by the different models. The human cvaluation aimed to capture important aspects summarization, including fluency, readability, and relevance.
Hongbo Zhang, Chen Tang, Tyler Loakmana, Chenghua Lina, and Stefan Goetze. 2023. Cadge: Context- aware dialogue generation enhanced with graph- structured knowledge aggregation. arXiv preprint arXiv:2305.06294.
For the human evaluation, recruited group of expert annotators with strong background in biomedical research. The annotators were provided with a set of summaries generated by each model and were asked to rate them on a Likert scale rang- ing from to 5. The Likert scale allowed anno- tators to provide subjective assessment of the summaries based on their expertise and judgment. The four aspects evaluated in the human evaluation were as follows:
Jingqing Zhang, Yao Zhao, Mohammad Salch, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11333 PMLR.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:7904.09675.
Kun Zhao, Bohao Yang. Chenghua Lin, Wenge Rong. Aline Villavicencio, and Xiaohui Cui. 2023. Evaluat- ing open-domain dialogues in latent space with next sentence prediction and mutual information. arXiv preprint 2335. 16967.
Fluency: Annotators assessed the language quality and coherence of the summaries. They considered factors such as grammar, sentence structure, and overall fluency of the generated text. Higher ratings on the Likert scale indicated better fluency.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang. Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2020. Graph neural networks: A review of methods and applica- tions. Al open, 1:57-81.
Readability: Evaluators focused on the readabil- ity and comprehensibility of the summaries. They assessed whether the generated summaries were clear, concise, and understandable to a non-expert audience. Higher ratings indicated better readabil- ity.
A Appendices
A.1 Implementation Details
Relevance: An important criterion was the rele- vance of the summaries to the original input doc- uments. Annotators evaluated whether the sum- maries captured the main ideas, key findings, and important concepts present in the source docu- ments. Higher ratings indicated greater relevance. Informativeness: Evaluate the extent to which the generated summaries provide sufficient and mean- ingful information derived from the given input. Assess the comprehensiveness and completeness
ChatGPT Prompts The performance of Chat- GPT is highly reliable on the quality of input prompts. We manually design and test prompts of abstract summarisation, and select the best cases the experimental results.
The Gaussian kernel smoothing used in Figure 4 is implemented with the coussian_fittell function from the python package of sdipy.n The ROUGE score evaluation is implemented
Others

---

Index
Target
Template
1
Generate Summaries
Write a summary according to given paper content, which is part of a medical scientific paper (A), The length of generated summary is expected to be larger than 130 words and less than (MAX, 13S, F.N words. \m/n The paper content of (A) is: (DOC') Arel Output:
2
Generate the Human
Write detailed humam evaluation guideline based the following content: (X)
Evaluation Guideline
Table The examples of ChatGPT prompt templates. The variable in the brackets () will be replaced by the actual text during the utilisation.
Model
BerS-Pr
BertS-Ri
BertS-F17
BartS
search in citation network-enriched text summari-
Golden
100
100
100
-0.26
sation:
ChatGPT
86.46
84.71
85.56
-3.11
and we have to wait for more advanced GPU de- vices or more optimised models (e.g. distilled mod- els) to make training those models to be practical. Currently, there are two main direction to further improve our citaion networks enhanced text sum- marisation: (1) The development of more efficient neural network that can effectively incorporate the graph- features derived from citations (Tang et al., 2023a; Yang et al., 2023a). (2) The iden- tification and extraction of key information from both the input document and its associated citations to enhance language understanding (Huang et al.. 2022; Tang et al., 2022c). We defer the exploration of these directions to future research endeavors.
Pubmed-I.ED
82,91
82.83
82.85
-3.37
SaameddPEGASSS
82.55
81
81.97
-3.42
Pubmed-BARE
84.26
82.49
83.35
-3.38
Pubmed- BART
Bhbmed-BARTT
one citation
84,20
82.54
83.34
-3.36
- citation aug.
84.33
82.92
83.60
-3.35
Table 6: BeS and BaS denote of BERTScore and BartScore, respectively. P. R. F1 represents the pre- cision, recall and Fl values, respectively.
of the summary. Consider the inclusion of impor- tant details and relevant facts.
By utilising a Likert scale with a range to 5, we were able to capture nuanced evaluations from the annotators. This human evaluation provided valuable insights into the overall performance of the models from the perspectives of fluency, read- ability, and relevance, allowing to gain deeper understanding of their summarization capabilities in the biomedical domain.
A.4 Future Works
In this paper, we propose novel framework de- signed to enhance the performance of biomedi- cal text summarisation using pre-trained language models. Recent years have witnessed the emer- gence of increasingly potent open-source language models, exemplified by Llama 27 and Baichuan" 18 However, the practical implementation of our ap- proach on these immensely large-scale models has been constrained by high computational demands. Consequently, we anticipate the need for more ad- vanced GPU hardware optimised models, such distilled models (Yang et al., 2023b), to render the training of these models feasible. Presently, there are two primary ways for advancing our re-
"https: //ai. meta. com/ com/ wwtm///am/.....m///// Whttps: t/w..hb..tttutt//////////e ctm/ wawaaaichhaannnnaiiiiiiee


Reviews:
[{"review_id": "UkYa8kRA0s", "reviewer": "Reviewer_AoVu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for your thoughtful and positive review of our paper. We also appreciate your detailed evaluation and your recognition of the contributions and merits of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a great work which reflects the idea that for scientific document summarization, the model should have some background knowledge, specially for biomedical documents. And incorporating the abstracts of the cited papers it provides enriched features (background information) for the LLMs. Furthermore, the authors have introduced a large corpus for biomedical document summarization. Furthermore, they have achieved the state-of-the-art performance, ", "reasons_to_reject": "From my side there is no reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper has 2 major contributions. Firstly, it has introduced a large scale biomedical research article summarization corpus. Secondly, it has proposed a novel approach to improve the performance of the LLMs for biomedical research article summarization. It's true that research articles can't express the whole thought in a standalone manner due to page limit restriction. To support the claims, they need to use citations. And to get proper knowledge enriched summaries, it is required to have the background knowledge which can be obtained via the citation network. This paper has explored this aspect fully and they have achieved state-of-the-art performance.  ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "pnTLZ2YTAe", "reviewer": "Reviewer_bXTQ", "summary": "", "questions": "A. It would be helpful if the authors could elaborate more on the lower precision scores of the best-performing method.\n\nB. Could the authors discuss more on how the longer sequences are handled during the encoding process?\n\nC. As the source articles could cite related work for different purposes (e.g., discussing background information, comparing the method, using the dataset, etc). Semantic Scholars are also adding this citation function information to their database. Due to the different purposes for the citation, the relevance or importance of the cited articles might be different in regard to the source articles. I wonder if adding such citation function information to the encoders would also help to decide the importance of the cited papers?\n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and insightful comments and we present our responses below.\n\nReject_1: In our approach, for each citation paper, we concatenate its abstract with the original article and then feed it to the encoder to produce an encoding (i.e., for N citations papers, there will be N encodings produced). The resulting encodings are then aggregated into a fixed size vector for the decoding process, as shown in Equations (3) and (4). Therefore, the input sequence is fixed and does  not grow linearly with the number of citation papers. We will make sure to make this more clear in our final paper version.\n\nReject_2: The lower precision scores  might potentially be attributed to a few factors .   First of all, the summaries generated by the baselines tend to contain more repeated words used in the ground truth leading to higher precision, but the actual quality of the summary is not higher, as evidenced by the lower scores for metrics of readability, bert/bart scores, and human evaluation in Table 2 and 3. In addition, our model incorporates more novel external phrases sourced from the citation papers, which have not been used in the ground truth, and hence has some impact on the precision. \n\nQuestion_A: Same as Reject_2 .\n\nQuestion_B: Same as Reject_1. In our approach, we conceptualise these citations as discrete nodes within a citation graph, thereby ensuring that the length of the input sequence remains invariant as the number of nodes (i.e., citations) increases. \n\nQuestion_C: This is a very good suggestion, and we believe the citation function information will be a very useful source of information. For instance, we could consider placing more emphasis on the citations whose purpose is for discussing background information, which will be more useful for feature enhancement. We will explore this in our future work.\n\nTypos Grammar Style And Presentation Improvements: All has been done, and we will put the literature section to the main content given an extra page allowed in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Biomedical text summarisation is an important one to the NLP community. Developing methods to improve the current models would be potentially beneficial for many downstream applications in this field.\n\nIt is a nice idea of including external knowledge from the abstracts of cited articles.\n\nThe developed dataset would be useful for the community.", "reasons_to_reject": "Combining the cited abstracts and original content of source papers together could make the input sequence very long for the encoders. It is not very clear how the longer sequences are handled during the encoding process, which might affect how the external knowledge from the cited papers is utilized.\n\nBased on the reported results, the precision scores of the best-performing model (citation agg) seem to be lower compared to other methods across the three evaluation methods (as reported in Table 2). ", "questions_to_authors": "A. It would be helpful if the authors could elaborate more on the lower precision scores of the best-performing method.\n\nB. Could the authors discuss more on how the longer sequences are handled during the encoding process?\n\nC. As the source articles could cite related work for different purposes (e.g., discussing background information, comparing the method, using the dataset, etc). Semantic Scholars are also adding this citation function information to their database. Due to the different purposes for the citation, the relevance or importance of the cited articles might be different in regard to the source articles. I wonder if adding such citation function information to the encoders would also help to decide the importance of the cited papers?\n\n\n", "paper_topic_and_main_contributions": "This paper incorporates the abstracts of cited papers into the content of source articles for biomedical literature summarization (CORD-19). Specifically, they construct a biomedical literature dataset, which contains the citation graphs and content of both cited and source papers; they revised the attention mechanism during the encoding process to measure the importance between original source paper content and abstracts of cited papers; they conducted a series of experiments to evaluate the performance of the proposed methods. The results show that their approach was able to have better performance than the baseline models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 321: are unable -? able?\n\nline 336-337: we were able to use...as the base language model to fine-tune on our dataset: do you mean unable to do so? as ChatGPT is close-source?\n\nTable 2 - w citeion agg: w citation agg\n\nI suggest the authors include the literature review section to the main content of the paper."}}, {"review_id": "IL8X8TH9Gm", "reviewer": "Reviewer_u9sN", "summary": "", "questions": "Please answer the queries stated in Reason to Reject section.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and insightful comments and we present our responses below.\n\nReject_1: We would like to clarify that in our approach, for each citation paper, we concatenated its abstract with the original article and then fed it to the encoder for producing an encoding (i.e., for N citations papers, there will be N encodings produced). The features of the resulting encodings of the concatenated inputs are aggregated into a fixed size vector for the decoding process, as shown in Equations (3) and (4). Therefore, the input sequence is fixed and does not grow linearly with the number of citation papers. To be more specific, the length limit ratio of the content of the original article and the abstract of one citation paper to the input is set by 3 to 1 in our implementation, where 80% paper abstracts in the corpus have the words less than the limit and can be fully incorporated. We will make sure to make this clear in a final paper version by providing more implementation details in the appendix. We are happy to provide further info as the reviewer needs.. \n\nReject_2: We thank the reviewer for raising this interesting point. In our methodology, we aim to ensure the quality of the cited article information that is utilised during generation via the incorporation of the attention-based mechanism as described from line 242 to 270. This mechanism essentially assigns a score to each cited article in the knowledge aggregation process and, by training it in an end-to-end fashion, the idea is that this component will learn to capture aspects such as the correlation between the source and citation article. We believe that the improvements demonstrated across both automatic and human evaluations suggest the component can effectively  learn such information, but also  agree that a more explicit modelling of such  relationships between the source and citation articles could offer further benefits to the knowledge aggregation process. We leave this to our future work.\n\nReject_3: Thanks for your suggestion! We would like to clarify the input (having citation information) of models \"- w one citation\" and \"- w citation agg.\" are actually different from that (having no citations) of the baseline \"Pubmed-BART\", so we employ the Mann-Whitney U test rather than the Wilcoxon signed-rank test. We are happy to provide further info as the reviewer needs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. a large-scale dataset is a good contribution to summarization space especially in biomedical domain where such large scale datasets are very sprasely available\n\n2. an interesting approach to enhance the knowledge learnt for decoding step to produce better relevant abstract.\n\n3. an in-depth analysis and wide variety of baselines considered in the study to show improvements using the proposed strategies.", "reasons_to_reject": "1. although the choice of models seems fine at first, but I am not sure as to how much of the citation information is actually being utilized. The maximum input size can only be 1024 and given the size of articles and the abstracts I am not sure how much inforamtion is being used in either. Can you comment on how much information is lost at token-level that is being fed to model.\n\n2. I like the idea of aggregation using various cited articles. The only problem and I might be possibly confused as to how are you ensuring the quality of chosen articles. It could be so that the claims made in the article might also be contradicting to the claims made in cited articles or might not at all be related to the claims being discussed in the articles. Do you have any analysis on correlation between cited articles and the main articles, and whether it affects the quality of generation?\n\n3. The authors have reprt significane testing but I think the choice of test might be incorrect. Since the comparision is to be done between two samples generated from same input why not some paired test setting was used like wilcoxon signed ranked test?", "questions_to_authors": "Please answer the queries stated in Reason to Reject section.", "paper_topic_and_main_contributions": "The authors propose a large-scale dataset to improve the state of abstractive summarization in biomedical domain. In particular they use the abstracts of citations of an article and the source article and propose a novel aggregation strategy to efficiently utilize the knowledge of the citation network and then generating refernce summary as the absract.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a method to generate summaries using abstracts of citations of an article and the source article by the proposed aggregation strategy based on the graph-styled citation network. They also create a large-scale dataset for abstractive summarization in the biomedical domain. The experimental results on their created dataset show that their proposed method outperformed baselines. The authors answered reviewers' questions regarding input to the model and cited articles in detail. After the rebuttal, the reviewers agreed with their score towards accepting the paper; thus, there is no reason to unfollow their decision.
Decision: Accept-Main