Enhancing Biomedical Lay Summarisation with External Knowledge Graphs
Authors: Tomas Goldsack
Zhihao Zhang
Chen Tang
Carolina Scarton
Chenghua Lin
Venue: EMNLP 2023 Main (2023)
Keywords: Summarisation
Knowledge Graphs

Abstract: Previous approaches for automatic lay summarisation are exclusively reliant on the source article that, given it is written for a technical audience (e.g., researchers), is unlikely to explicitly define all technical concepts or state all of the background information that is relevant for a lay audience. We address this issue by augmenting eLife, an existing biomedical lay summarisation dataset, with article-specific knowledge graphs, each containing detailed information on relevant biomedical concepts. Using both automatic and human evaluations, we systematically investigate the effectiveness of three different approaches for incorporating knowledge graphs within lay summarisation models, with each method targeting a distinct area of the encoder-decoder model architecture. 
Our results confirm that integrating graph-based domain knowledge can significantly benefit lay summarisation by substantially increasing the readability of generated text and improving the explanation of technical concepts.

Enhancing Biomedical Lay Summarisation with External Knowledge
Graphs
Tomas Goldsack', Zhihao Zhang,, Chen Tang', Carolina Scarton', Chenghua Lin
DDpartment of Computer Science, University of Sheffield, UK
CCollege of Economics and Management, Beijing University of Technology, China,
3 Department of Computer Science, The University of Surrey, UK
4 Department of Computer Science, The University of Manchester, UK
(tgoldsack1, c.scartonjesheffieldd ac. uk zhhzhang@bjut. edu.cm
chen. tangenurgrurry...o
tteenghu.,Iinmancceee www.nnnnheeeee..c..mm ac.uk
Abstract
Lay summary
Previous approaches for automatic lay sum- marisation exclusively reliant on the source article that, given it is written for technical audience (e.g., researchers), is unlikely to ex- plicitly define all technical concepts or state all the background information that is rele- vant for lay audience. We address this issue by augmenting eLife, existing biomedical lay summarisation dataset, with article-specific knowledge graphs, each containing detailed information on relevant biomedical concepts. Using both automatic and human evaluations, systematically investigate the effectiveness of three different approaches for incorporating knowledge graphs within lay summarisation models, with each method targeting a distinct area of the encoder-decoder model architec- ture. Our results confirm that integrating graph- based domain knowledge can significantly ben- efit lay summarisation by substantially increas- ing the readability of generated text and im- proving the explanation of technical concepts."
Summarisation Model
"allogeneic"
Allogeneic
Trom toe bntriaat
Article spocific Kngwledge Graph
Article text
Figure 1: Overview of the "knowledge graph-enhance Lay Summarisation' task formulation, exemplifying graph external information.
is unlikely explicitly include all the information necessary for the lay summary, such background details or definitions. As a result, lay summaries are often highly abstractive, adopting a simpler lexi- con than the original article (Goldsack et al., 2022). and are typically written by experts who possess the knowledge required to simplify and explain the contents of the article (King et al., 2017).
1 Introduction
Lay summarisation consists of generating concise summary that illustrates the significance of longer technical (or otherwise specialist) text and is com- prehensible the non-expert (Kuehne and Olden, 2015). A lay summary should contain minimal jargon and technical details (e.g.. methodology). instead focusing largely on the simplification of key technical concepts and the explanation or rele- vant background information, thus allowing readers without technical knowledge to grasp the general topic and main ideas of an article (Srikanth and Li, 2021; Goldsack al., 2022). However, since the original article is intended for technical audience who already possess some domain knowledge,
Despite this disparity between lay summary and source article, automatic approaches to lay sum- marisation have typically relied solely upon the source article as input (Chandrasekaran et al., 2020; Guo et al., 2021; Luo et al., 2022a). Aiming to address this, we propose enhancing lay summari- sation models with external domain knowledge, conducting the first study on knowledge graph- enhanced lay summarisation with focus on biomedical articles. We augment eLife (Goldsack et al., 2022), an existing high-quality lay sum- marisation dataset, with article-specific knowledge graphs containing information on the technical con- cepts covered within the articles and the relation-
Corresponding author /Our code and data is available at https: // t/github..com wtt.cooom/ooo.eee dsack1 SEnnanccing_iiioee cal_ Lay_ Swwmarisatttttm th_ External Knwwledge @rnwlesgeeGraahs...

---

ships between them (exemplified in Figure 1), thus providing a structured representation of the domain knowledge that an expert human author might draw upon when writing a lay summary ($3). In do- ing this, we hypothesise that a model's is ability to simplify and explain technical concepts for lay audience will improve.
journals, respectively). 2 Both studies benchmark their datasets with widely-used summarisation ap- proaches, with BART variants (Lewis et al., 2020) invariably achieving the strongest performance. another highly related work, Luo et al. (2022a) address the task of readability- summari-- sation using data derived from PLOS, training a BART-based model to produce both the abstract and lay summary of an article in controlled set- ting.
Although other forms of summarisation task (e.g., news articles) have seen significant research in the enhancing models using knowledge graphs (Huang et al., 2020; Zhu et al., 2021a; a; Lyu et al.. 2022), this has yet to be explored for lay sum- marisation. To our knowledge, there is no work on determining the most effective way to incorpo- rate graph-based knowledge for lay summarisation other summarisation tasks. Therefore, tematically investigate the effectiveness of three different methods for injecting graph-based in- formation into lay summarisation models ($4) assessing them with both automatic and human evaluation ($5 & $6). Our results demonstrate that the integration of graph-based domain knowledge can significantly improve automatic lay summari- sation enabling models to generate substantially more readable text and to better explain technical concepts.
In contrast to previous works, we investigate the unexplored approach modelling and inte- grating structured domain knowledge into lay sum- marisation models using article-specific knowledge graphs.
sys-
2.2 Knowledge Graph-Enhanced Text Generation
In recent years, the utilisation of knowledge graphs (KGs) containing external knowledge for text gen- eration has seen increased interest, particularly when it comes to the modelling of commonsense knowledge. In particular, works focusing on tasks such as dialogue generation (Zhou et al., 2018; Tang et al., 2023), commonsense-reasomis et al., 2021), story generation (Guan et al., 2019; Tang al., 2022), essay generation (Yang et al., 2019) have all seen the introduction of common- sense KG-enhanced models.
(Liu
2 Related Work
2.1 Lay Summarisation
Some recent work has also focused on using KGs for abstractive summarisation, but tending towards modeling internal knowledge. Aiming to improve the faithfulness and informativeness of summaries, Huang et al. (2020), Zhu et al. (2021a), and Lyu et al. (2022) all utilise OpenIE to construct fact- based knowledge graphs from source documents (news articles). Huang et al. (2020) and Zhu et al. (2021al 1a) extract graph node features using graph attention networks (Veličković et al., 2017), before incorporating these into the summarisation model decoder using an attention mechanism. Lyu et al. (2022) instead make use of additional semantic loss measures to attempt capture extracted facts using an adapted pointer-generate network.
The task of lay summarisation is relatively novel one, introduced by the LaySumm subtask of the CL-SciSumm 2020 shared task series (Chan- drasekaran et al., 2020). Introducing a multi- domain corpus of 572 article-lay summary pairs, the task attracted a total of participants. The win- ning system, proposed by Kim (2020), adopted a hybrid approach, using PEGASUS-bert based (Zhang et al., 2020) model to generate an initial abstractive lay summary before augmenting this with suffi- ciently readable article sentences extracted by BERT-based model (Devlin et al., 2019).
Subsequent lay summarisation work has focused almost exclusively on the introduction and bench- marking of new corpora (all from the biomedical domain), rather than introducing specific modelling approaches for the task. Guo et al. (2021) intro- duce CDSR, dataset derived from the Cochrane Database of Systematic Reviews, whereas Gold- sack et al. (2022) introduce PLOS and eLife, two datasets derived from different biomedical jour- nals (the Public Library of Science and eLife
In contrast to previous works, we apply KG- based techniques to biomedical lay summarisation (as opposed to news article summarisation), a do- main with additional challenges, including the ex- tensive length of input articles and the presence of
?A version of these datasets with different test sets is also used within the Riol ay 2023 shared task (Goldsack et al., 2023a), that ran in parallel with this work.

---

complex technical concepts that need be simpli- fied or explained. Furthermore, due to the unique requirements of the task, we innovate by construct- ing knowledge graphs largely using external do- main knowledge sources rather than from the arti- cle itself, as is the case with previous approaches.
Metric
Abstract
Lay Summary
Definitions
FKGL.
15.57
10.92
10.55
CLI DCRS}
17.68
12.51
13.02
11.78
8.83
10.36
WordRank.
9.21
8.68
8.6
Table 1: Mean readability scores for abstracts, lay sum- maries, and key UMLS definitions for eLife. FKGL Flesch-Kinkaid Grade Level, CLI Coleman-Liae Index, DCRS Dale-Chall Readability Score.
3 Article Knowledge Graphs
We augment eLife, an existing dataset for biomedi- cal lay summarisation with heterogeneous article- specific knowledge graphs (KGs). Each KG contains structured information the complex biomedical concepts covered within the article and the relationships between them. In order localise this information and provide an indication of where in an article a concept is mentioned, we also choose to model the article's s section- based document struc- ture within our graphs through the use of section- specific nodes. In the following, we describe in detail: 1) our process for extracting the knowledge that is used by our model ($3.1), and 2) how structure that knowledge within knowledge graph ($3.2). The methods by which we integrate graph- based knowledge into summarisation models are discussed in $4.
cles in eLife, we apply MetaMap to each section in turn, retrieving all mentioned UMLS concepts. We restrict MetaMap to only a select number of English vocabularies without prohibitive access strictions, but otherwise run using default set- tings.
In line with observations made in previous works (Lai et al., 2021), we found that MetaMap, whilst succeeding in linking biomedical entities men- tioned in the text with their corresponding UMLS concepts, also frequently returned a number of irrelevant concepts. Therefore, we adopt a text overlap-bane based approach to filter the original pool of extracted concepts for given section, which we empirically found to eliminate the vast majority of the noise.
3.1 Knowledge Extraction
For each remaining UMLS concept, we retrieve all semantic types with which it is associated, in addition to the formal definitions both the con- cept and semantic types. Notably, these definitions are used as an integral component within all three KG-enhancement methods." 4 In order to confirm their suitability for lay audience, we calculate and compare their average readability scores with those reported by Goldsack et al. (2022) for both the technical abstracts and lay summaries of eLife articles. The results of this analysis, given in Table 1, show that the UMLS definitions obtain scores that are overall much closer to those of the lay summaries than the abstracts, actually exceeding them in two out of the four metrics (FKGL and WordRank). An example of the definition format used for text augmentation is given in Figure 6 in the Appendix. In the next section, describe how we represent all extracted information within article-specific knowledge graphs.
To extract relevant domain knowledge for an article, we draw upon the Unified Medical Language Sys- tem (UMLS) (Bodenreider, 2004). This rich and actively- maintaly resource has long been used as a key knowledge source for NLP in the biomedical domain (McCray et al., 2001; Demner-Fushman et al., 2010; Kang et al., 2021) and is comprised of three primary components: the Metathesaurus, the Semantic Network, and the Specialist Lexicon and Lexical Tools. The Metathesaurus is extensive multi-lingual vocabulary database containing infor- mation on a large number of biomedical concepts, including their various names and definitions. The Semantic Network defines a set of semantic types that represent broad subject categories into which all concepts in the Metathesaurus can be assigned. Additionally, high-level relationships that occur be- tween different semantic types are also defined. To extract the UMLS concepts mentioned within given article, we utilise MetaMap (Aronson and Lang, 2010), one of the Lexical Tools provided alongside UMLS for this exact purpose, that is widely used in previous work (Sang et al., 2018; Sharma et al. 2019: 9; Lai et al., 2021). ). For all arti-
MMore details on MetaMap vocabularies, noise reduction process, and average article KG statistics are provided in the Appendix.
'Concepts without a formal UMLSS definition are also re- moved from the final pool.

---

eiter vi_Sect
lg antibody
Cell Component
SIOES
eitee vi_
Viral
Organism
eifc
CCTONEE
S1835-
THPO WE
STINGL Gena
Materiak
infeclinns disens
Maroase restrictinn fector maghhggs
GenoRe
Figure 2: Example of article knowledge graph structure. Graph nodes are coloured as follows: Metadata, Concept, Semantic type.
Document,
Section.
3.2 Graph Construction
Semantic type node - nodes representing semantic types from the Semantic Network. These are labelled with their unique Semantic Type identifier (TUI).
Each graph G = {V,E), where V is a set of nodes (or entities) and & is set of edges. Each edge eij E & defines a relation Tiy between entities 0i, My C V, and thus be represented as triplet eij - (w, Tijy uz). All graphs heterogeneous, containing multiple types of entities and relations. Figure 2 presents visualisation an article knowl- edge graph. Each type of node is described below:
In addition the 54 different relationship types defined within the semantic network (e.g., affects in Figure 2), we define several relations order to represent the graph structure and additional meta- data. Specifically, we define the relations contains, was crus_published_ie inn, has has_lille, and has keyword.
Document node - the central root node, which is the ancestor of all other nodes in the graph. We label this node simply with the
4 Knowledge-Enhaneed Lay Summarisation Approaches
unique ID assigned to each article.
We investigate the effectiveness of three differ- ent methods for incorporating external knowledge from article graphs into encoder-decoder-bbnn basedd summarisation models. Our experiments are care- fully designed as to target distinct aspect within the model architecture (i.e., the input, the coder, and the decoder) with each selected method, taking inspiration from models that have recently been proven effective the domain of news ar- ticle summarisation (Zhu et al., 202 la; Pasunuru et al., 2021). Figure provides a visualization of how each of these approaches fits into this architec- ture. To allow the ingestion of the full input arti- cle. we make of Longformer Encoder-Decoder (Beltagy al., 2020) as our base model for all experiments. This BART-based model replaces standard transformer self-attention with sparse attention mechanism that scales linearly to the se- quence length, enabling the processing of longer
Section node each section node represents specific titled section (e.g. Introduction) of the document, including the abstract. To label these nodes, we concatenate the article ID with " "_Abs" for the abstract or " "_Sec(i)" for other sections, where {i} is the index of the section (zero-based).
Metadata node identify additional informa- tion relating to the article or its specific sec- tions. This includes article and section titles, article keywords, and the date of publication.
Concept node nodes representing UMLS concepts. These are labelled with their unique UMLS identifier (CUT).
"Note that, for visual clarity, this example contains signifi- cantly fewer concept and semantic type nodes than are present in the actual article graph.

---

A
Output Probahilitin
B
Graph Embedding
Document Embedding
X
L
Decoder
Food Fwd a Add & Norm
Norm
Graph Attn Net
Encoder
Cross Altn Add & Norm
Graph
Outputs (shifted right)
Article Knowledge Graph
Article
Visualisation of how various Knowledge-enhanceemed approaches incorporate external knowledge from article knowledge graphs into transformer-bane based encoder-decoder architecture (as described in $4). A) Decoder cross-attention, B) Document embedding enhancement, C) Article text augmentation.
Figure 3:
texts (such research articles). We describe each knowledge-nnhancemeett approach in detail below.
Specifically, we concatenate the document and graph embeddings, before passing them through additional encoder layer. For given input document X, this process can be formalised as follows:
(A) Decoder cross-attention. We make use of a Graph Attention Network (GAT) (Veličković et al.. 2017) to obtain an embedding of the article graph g in parallel with the base model encoder.
H Encoder(X)
(2)
H = GAT(G)
(1)
HC [H HD1
(3)
These Graph Neural Network (GNN) models pro- duce a final set of node features (i.e., graph em- bedding) by using attention layers to efficiently ag- gregate over the features of neighboring nodes, and are widely used in current literature to aggregate graph-based information for NLG tasks (Huang al., 2020; Zhu al., b021b; Liu al., 2021). During the decoding phase, we follow previous works (Zhu et al., 2021a) by forcing our model to attend to the KG embedding H1S Specifically, in every transformer layer of the decoder, we include a second cross-attention mechanism that occurs directly after the standard encoder cross-attention (sce arrow A in Figure 3) and attends to the output of the GAT-based model.
H" = Se.p--noddeLLayrr(( (H (1-p) Hº (4)
where H-X an embedding of document X, HC is the concatenated document and graph embeddings. H is the final 'enhanced' document embedding that is subsequently attended to during decoding, and is a scaling factor controlling the extent to which the additional encoder layer output is incor- porated in the final enhanced document embedding. Note that p is treated as a hyperparameter the model, for which value of 0.25 was found provide the strongest validation set performance."
(C) Article text augmentation. We also experi- ment with simply augmenting the input text with textual explanations of the key concepts (and their relations) derived from the graph. Whilst this may be arguably the most 'natural' way for PLM- based model to interpret external information, this approach leads
(B) Document embedding enhancement. Again, obtain embedded graph represen- tation HS using the GAT model, but rather than attending to graph embeddings during decoding, we follow Pasunuru al. (2021), combining the embedded node information into the final docu- ment embedding (i.e., the output of the encoder).
exponential increase in the
"We also find that large value of causes significant degradation in performance, suggesting that the original docu- ment information is lost.

---

number of tokens required describe each ele- ment, hence it restricted set of few concepts. We select only those entities which are likely to be most central to the topic of the article (and, therefore, relevant to the lay summary). Specifi- cally, we take the concept nodes that are mentioned in the article abstract and use the graph relations and retrieved definitions to provide textual expla- nation of these salient concepts and their seman- tie types, which is then prepended to the article text. This takes the format of "/concept_namet = (concept_definitionn definition). cconcept_namer is a (sem- type "ype_aame."" repeated for each selected concept, followed by semantic type definitions formatted "[semtype_nane _anme]==_=______e [semtype_ddeeeee _definitionn" repeated for all mentioned semantic types.
5.3 Implementation and Training
Each knowledge-based approach is implemented by manually adapting the Longformer implementa- tion from Huggingface (Wolf et al., 2020) and, fol- lowing previous work on lay summarisation (Chan- drasekaran et al., 2020; Luo et al., 2022b; Gold- sack et al., 2022), uses the full article text as input. For GAT-hased models, we make use of the Deep Graph Library package (Wang et al., 2019) to im- plement 3-layer GAT with 4 attention heads at each layer. For article graphs, we vary our node initialisation approach based node type (as de- fined in $3.2). Specifically, we initialise concept and semantic type node features, with the embed- dings of their textual definitions; document and section nodes with the embeddings of their title text (with title metadata nodes being subsequently ignored); and remaining metadata nodes (publi- cation date and keywords) with embeddings on their textual content. All embeddings are gener- ated using SciBert (Beltagy al., 2019), lan- guage model specifically trained on research pa- pers from Semantic Scholar (Ammar et al., 2018) that widely used for scientific data (Cohan et al.. 2019; Cai et al., 2022; Goldsack et al., 2023b). Furthermore, all node embedding features are con- catenated with one-hot features according to node type, as well as Random Walk Positional Encod- ings (Dwivedi et al., 2021). Following initialisa- tion with we train all models on A100 GPUs and retain the checkpoint with the best validation set performance (more details provided in the Appendix).
5 Experimental Setup
5.1 Data
We derive knowledge graphs for all articles eLife (Goldsack et al., 2022), a dataset for biomedical lay summarisation containing 4,828 article-summary pairs. Target summaries are expert- wertten lay sum- maries (i.e., summaries with a non-expert target audience) and inputs are the full text of the cor- responding biomedical research articles. As ex- plained in I we believe this task be particularly suitable for domain knowledge augmentation due to the contrast in the level of expertise of the target audience between source and target which causes a discrepancy in the language used (specifically, reducing or explaining jargon terms) and the level of background information required.?
checkpoint,
5,4 Evaluation Setup
5.2 Baselines
We conduct both automatic and human evaluations to provide a comprehensive assessment of how each
As a baseline model, we include BART (Lewis et al., 2020), the state-of-the-art benchmark re- ported by Goldsack et al. (2022) for eLife, as well as in other previous lay summarisation works (Guo et al., 2021). Additionally, we include the reported performance of BART BARTscaffoo fold (Goldsack et ul., 2022), a variant of BART trained to produce both the abstract and lay summary of an article in a controlled setting, which is equivalent to the model proposed by Luo et al. (2022a). 8
method affects the overall performance.
Automatic evaluation. For each model, we re- port the average scores of several automatic met- rics on the test split of eLife. As is common prac- tice, we report widely-used summarisation metries: BERTScore (Zhang al., 2019) and the Fl-scores of ROUGE-1, 2, and L (Lin, 2004).
"This discrepancy evidenced the analysis provided by Goldsack et al. (2022).
To assess the readability of generated summaries. we report Flesh-Kineal Grade Level (FKGL) and Dale-Chall Readability Score (DCRS), both of which compute an estimate of the US US-grade level
"Note that the original code for Luo et al. (2022a) is not yet available at the time of writing and their results are reported on a different datasct and thus are not comparable.

---

Relevance
Readability
Factuality
Model
R-1
R-21
R-L1
BeSt
CLI
DCRS}
BaS
BART
46.57
11.65
43.70
84.94
11.7
9.36
-2.39
BART fold
45.28
10.99
42.51
84.65
11.32
9.19
-2.57
Longformer
47.23
13.20
44.44
85.11
11.72
9.09
-2.56
text-aug 48.58
14.24*
45.71
85.4"
10.94*
8.72*
-2.45*
-doc-enhance
48.
14.43*
45.52
85.33*
10.72*
8,50*
-2.35*
decoder-attn
48.30
13.93
45.45*
85.39*
10,99*
8,75*
-2.48*
Table 2: Average performance of models on eLife test split. R ROUGE FI, BeS BERTScore FI, CLI Coleman-Liat Index, BaS = BARTScore. denotes that KG-enbanced model results statistically significant with respect to the base model (Longformer) by way of Mann- Mann-Whitney U test.
required to comprehend text."
largely find to be detrimental to model perfor- mance. Discussion and results (Table 6) of com- bined methods are provided in the Appendix. We discuss the performance of individually applied methods below, focusing on each aspect of auto- matic evaluation turn.
Additionally, we evaluate factuality using BARTScore (Yuan et al., 2021), which has been shown have a strong aligmment with human judg- ments of factual consistency in a recent study fo- cusing specifically on long documents (Yee Koh et al., 2022). Following Yee Koh et al. (2022), we adapt BARTScore use Longformer (thus allow- ing it to process the entire document input) and fine-tune on eLife.
Relevance Longformer can be seen to outper- form the standard BART model in terms of rele- vance metrics, indicating that processing the entire document provides some benefit for lay summari- sation. Additionally, all three knowledge enhance- ment methods significantly obtain improved scores across almost relevance metrics (with the excep- tion of R2 for the 'decoder attention' model). This provides a strong indication that the addition of graph-based domain knowledge provides models with relevant external information, enabling them to produce lay summaries that closer in resem- blance the high-quality references.
Human evaluation. To provide comprehensive assessment of the summaries generated by each knowledge-enhanced model, we conduct a human evaluation focusing on readability and factuality. Specifically, making use of 5 randomly sampled ar- ticles from the eLife test set, we ask human judges to evaluate each sentence within generated sum- mary along the following binary criteria: 1) Factu- ality- the sentence factually correct (with respect the source article); and 2) Readability would a layperson be able to understand this sentence. 10 To help determine the factuality of the sentence, the annotator has access to the PDF of the source article as well as the reference lay summary." !I
For readability metrics, it can first be noted that Longformer-based models obtain lower CLI and DCRS scores than those BART- based models. The calculation of CLI is based on the number of characters. words, and sentences it contains, whereas DCRS is based on the frequency of "familiar" (i.e., commonly-usee words, suggest- ing that Longformer produces summaries that are less syntactically and lexically complex.
Readability
6 Experimental Results
6.1 Automatic Evaluation
Table 2 presents the performance of different mod- els using the described automatic evaluation met- rics the test set of eLife. In addition to applying KG-enhancement methods in isolation, we also ex- periment with combining different methods, which
We observe that the application of all knowledge enhancement methods results in improved scores for both metrics, with the document enhancement approach achieving the largest gains. This indicates that all knowledge enhancement methods able successfully influence the phrasing and structure of the summaries being generated by increasing the usage of more common (i.e., less technical) termi- nology. As reported Table 1, the average CLI and DCRS scores for the reference lay summaries
"Computed using the textstat package. 10 An average of 68.5 sentences evaluated per model. "Following Yee Koh et al. (2022), we encourage judges to use text-based search within the article to quickly identify relevant passages rather than asking them to read each article in full, reducing the cognitive burden placed upon them.

---

Model
#
Readability
Factuality
[Melosis]
Longformer
73
78.08
60.96
glabrous skin of the I Longformer During metosis, the DNA in one of the chnumo- somes is cupied and then the two copies are recombined so that each new generation will have a single copy of the gene that en- codes the protein encoded by that gene." 1/2 w/ text-aug ... process known meiosis
text-aug
65
96.92
68.46
doc-enhance
67
97.01*
55.97
decoder-attn
69
95.65*
63.77
copics of each chromosome are then exchanged between the newly formed cells. which resalis in unique sel genes being passed generation. 2/2
next
Table 3: Average percentage of generated sentences pos- itively classified by judges for each high-level binary criteria. # total number of sentences generated across all summaries. * ddnooees that KG-enhanced model re- sults are statistically significant with respect to the base model (Longformer) by way of Mann- Mann-Whitmey U test.
[Glabrous skin Mechanoreceptors]
Longformer - The orientation of an object depends largely on how its edges activate mechanoreceptors
fingertips. 0/2
w/ doc-enhance The fingertip's surface is covered by a layer skin known glabrous skin. 2/2 These cells responsible for sensing touch, they are responsible for detecting orientation of objects that touch them, 2/2
of eLfee are 12.51 and 8.83, respectively.
[Slow wave sleep]
Factuality For BARTScore (BAS), we again see a statistically significant improvement over the base Longformer model for all KG-enhancemmen methods, with the greatest improvement being ob- tained by the doc-enhance method. In order to gain further insight in these results, we also calculate the BARTScore values obtained by reference sum- maries, getting a mean score -2.39, which similar to that of all tested models (and identical to that of BART). This suggests that all models are able to produce summaries with generative probabilities similar to that of the reference summaries. How- ever, given that one model (doc-enhance) actually outscores the reference summaries, further analysis is needed to gain an understanding of the difference in the factual correctness of summaries produced by each method, for which we turn to our human evaluation.
Longformer- - Most studies sleep have focused on slow wave sleep, in which brain's is activity alternates between periods alternating periods of slow and fast %/2
0/2
w/ decoder-attn Slow wave sleep is characterized by rhythmic waves of electrical activity in the brain, which are thought to be part of the process by which the brain consolidates memories. 2/2
Figure 4: A case study comparing how applica- tion of each method affects the explanation of specific technical concepts within the human evaluation sample. Colours and superscript are used denote the number of evaluators who judged the sentence as readable for lay audience (e.g.. 2/2 = 2 out of 2 evaluators).
ing significantly higher than the base Longformer model. Alternatively, the results for factuality show that, although there is a slight variance in perfor- mance between KG methods, none of them are judged to be by statistically significant margin. These results indicate that all methods are able effectively introduce relevant external information into the model, enabling to produce text that is casier for a lay audience to comprehend without significantly compromising the factual correctness of the base model.
6.2 Human Evaluation
Given the challenging and time-consuming nature of evaluating the factuality of technical biomedical sentences against the source article, we carefully plan our human evaluation so as to ensure relia- bility in our results. We employed two annotators to evaluate generated sentences following the pro- cedure laid out in $5, both of whom are experts in NLP and familiar with common model short- comings (e.g., hallucinations). Table 3 presents the total percentage of sentences that were positively classified for both readability and factuality aver- aged across evaluators, who achieve a Cohan's 's k 0.42.
Case Study To gain better insight into how knowledge enhancement methods influence the readability of generated summaries, we present a case study in Figure in which we compare the ex- planations of specific technical concepts generated by KG-enhanced models and the base Longformer model, alongside their annotator ratings. 12
These examples demonstrate how KG- enhancement methods improve the model's handling of technical concepts, thus making them easier to understand for a lay reader. Specifically,
The results in Table 3 suggest that the application of all KG-enhancement methods causes notable increase in the readability of the text produced by the model, with all models scor-
Discussion
An extended version of this case study is given in Figure 5 in the Appendix.

---

examples show how methods can influence the model generate an explanation in instances where the base model fails to provide one (b) or improve the explanation in instances where the base model's is difficult to understand (a and c).
References
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavat- ula, Iz Beltagy, Miles Crawford, Doug Downey, Ja- son Dunkelberger, Ahmed Elgohary, Sergey Feld- man, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Pe- ters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018. Construction of the litera- ture graph in semantic scholar. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies. Volume (Indus- try Papers), pages 84-91, New Orleans Louisiana. Association for Computational Linguisties.
7 Conclusion
This papers presents the first study on the use of knowledge graphs to enhance lay summarisa- tion, augmenting the biomedical lay summarisa- tion dataset eLife with article-spccific knowledge graphs containing domain-speciffe external knowl- edge on relevant technical concepts. We compare three distinct approaches for incorporating graph- based knowledge into encoder-decoder summarisa- tion models. placing an emphasis on the readabil- ity and factual correctness of the generated output. Our results suggest that integrating external knowl- edge has the potential to substantially improve lay summarisation, particularly for the generation of readable text and explanation of technical concepts. We would like see future work investigate the use of additional graph representations, as well as their integration into larger models that adopt different architectures (c.g., decoder-only).
Alan R Aronson and Frangois-Michel Lang. 2010. An overview of MetaMap: historical perspective and recent advances. J. Am. Med. Inform. Assoc., 17(3):229-22 236.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing 3620, Hong Kong, China. Association for Computa- tional Linguistics.
pages 3615-
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The donu-decument transformer.
Limitations
Olivier Bodenreider. 2004. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acidy Rex., 32(Database D2677)
One possible limitation of our work is derived from the use of resources from UMLS (i.e., UMLS con- cept names, semantic types and relations, defini- tions, etc.). Accessing these resources requires an individual license with the US National Library of Medicine (NLM), and their subsequent distribu- tion is restricted by this license agreement. There- fore, it likely that we will have to confirm the license status of those who wish to have access to the knowledge-grain In an attempt to reduce any potential impacts this will have on the ability to share our resources, we only make use of only select number of vocabu- laries less restrictive licences. More details on the vocabularies used are provided in the Appendix.
Xiaoyan Cai,, Sen Liu, Libin Yang, Yan La, Jintao Zhao, Dinggang Shen, and Tianming Liu. 2022. COVID- Sum: A linguistically enriched SciBERT-base sum- marization model for COVID-19 scientific papers. Journal of Biomedical Informatics, 127:103999.
Muthu Kumar Chandrasekaran, Guy Feigenblat, Ed- uard Hovy, and Anita de Waard. 2020. Overview and insights from the shared tasks at scholarly docu- ment processing 2020: CL-SciSumm, LaySumm and, In Proceedings of the First Workshop on Scholarly Document Processing, pages 214-224. unknown.
resources used in this work.
Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, and Dan Weld. 2019. Pretrained language models for sequential sentence classification. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 3693-3699, Hong Kong. China. Association for Computational Linguistics.
8 Acknowledgements
This work was supported by the Centre for Doc- toral Training in Speech and Language Tech- nologies (SLT) and their Applications funded by UK Research and Innovation [grant number EP/S023062/1J.
Dina Demner-Fushman, James Mork. Sonya Shooshan. and Alan Aaronson. 2010. Wumls content views appropriate for nlp processing of the biomedical lit- erature vs. clinical text. In Journal of biomedical informatics, 43(4), page 587-594.

---

Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
First Workshop on Scholarty Document Processing. pages 328-335, Online. Association for Computa- tional I. Linguisties.
Stuart R F King, Emma Pewsey, and Sarah Shailes. 2017. Plain-language Summaries of Research: An inside guide to eLife digests. eLife, 6:e25410.
Lauren M Kuchne and Julian D Olden. 2015. Lay sum- maries needed to enhance science communication. Proceedingy of the National Academy of Sciences, 112(12)3355-355
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Lau- rent, Yoshua Bengio, and Xavier Bresson. 2021. Graph neural networks with learnable structural and positional representations.
Tuan Lai, Heng Ji. ChengXiang Zhai, and Quan Hung Tran. 2021. Joint biomedical entity and relation extraction with knowledge-enhanced collective in- ference. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1tth International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 6248-6260, Online. Association for Computa- tional Linguistics.
Tomas Goldsack, Zheheng Luo, Qiangian Xie, Car- olina Scarton, Matthew Shardlow, Sophia Ananiadou, and Chenghua Lin. 2023a. Overview of the bio- laysumm 2023 shared task on lay summarization of biomedical research articles. In The 22nd Work- shop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 468-477 Toronto, Canada, Association for Computational Linguistics.
Tomas Goldsack, Zhihao 7hang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Cor- pora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 10589-10600 10604, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-, Online, Association for Computa- tional Linguistics.
pre-training
Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and
Carolina Scarton. 2023b. Domain-Driven and Discourse-Guided Scientific Summarisation. In Ad- vances in Information Retrieval, pages 361-376, Cham. Springer Nature Switzerland.
Chin-Yeu Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story Ending Generation with Incremental Encod- ing and Commonsense Knowledge. Proceedings of the AAAI Conference on Artificial Intelligence, (6001)66473-4880
Ye Liu, Yao Wan, Lifang He, Hao Peng. and Philip S. Yu. 2021. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. 35:6418-6422
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022a. Readability controllable biomedical docu- ment summarization. In Findings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, Abu Dhabi. Association for Computa- tional Linguistics.
Yue Guo, Wei Qiu, Yizhong Wang. and Trevor Co- hen. 2021. Automated Lay Language Summariza- tion of Biomedical Scientific Reviews. Proceedings of the AAAI Conference on Artificial Intelligence. 35(1):160-168.
Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summariza- tion with semantic-driven cloze reward. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5094-5107, On- line. Association for Computational Linguistics.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022b. Readability controllable biomedical docu- ment summarivation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4667 6667-4680, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Tian Kang, Adler Perotte, Casey Ta, and Chunhua Weng. 2021. Umls-based data augmentation for natural lan- guage processing of clinical research literature. In Journal of the American Medical Informatics Associ- ation JAIMA 28(4), pages 812-823, Online.
Yuanjic Lyu, Chen Zhu, Tong Xu, Zikai Yin, and En- hong Chen. 2022. Faithful abstractive summarization via fact-aware consistency consistency-constrainne constrained transformer. In Proceedings of the 31 ACM International Con- ference Information Knowledge Management, CIKM '22 page 1410-1419, New York, NY, USA. Association for Computing Machinery.
Seungwon Kim. 2020. Using pre-trained transformer for better lay summarization. In Proceedings of the

---

Alexa McCray, Olivier Bodenreider, James Malley, and Allen Browne. 2001. Evaluating umls strings for natural language processing. In Proceedings of AMIA Symposium, page 448-452.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Hug- gingface's 's transformers: State-of-thhe natural lan- guage processing.
Ramakanth Pasumuru, Mengwen Liu, Mohit Bansal, Su- jith Ravi, and Markus Dreyer. 2021. Efficiently sum- marizing text and graph encodings of multi-document clusters. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4768-4779, Online. Association for Computational Linguistics.
Pengcheng Yang. Lei Li, Fuli Luo, Tianyu Liu, and Xu Sun. 2019. Enhancing topic-to-essay generation with external commonsense knowledge. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2002-2012, Flo- rence, Italy. Association for Computational Linguis- tics.
Shengtian Sang, Zhihao Yang, Lei Wang. Xiaoxia Liu, Hongfei Lin, and Jian Wang. 2018. Sematyp: knowledge graph based literature mining method for drug discovery. BMC Bioinformatics, pages 1471 2105.
Huan Yee Koh, Jiaxin Ju, He Zhang, Ming Liu, and Shirui Pan. 2022. How far are we from robust long abstractive summarization? In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 2682-2698,, Abu Dhabi. Association for Computational Linguistics.
Soumya Sharma. Bishal Santra, Abhik Jana, Santosh Tokala, Niloy Ganguly, and Pawan Goyal. 2019. In- corporating domain knowledge into medical NLI using knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing Hong Kong, China. Association for Computational Linguistics.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems, volume 34, pages 27263-27277 Curran As- sociates, Inc.
TEMNLP-IJCNLPP pages
Neha Srikanth and Junyi Jessy Li. 2021. Elaborative simplification; Content addition and explanation gen- cration in text simplification. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 5 5133-5137, Online. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinherger, and Yoav Artzi. 2019. Bertscore: Evalu- ating text generation with bert.
Chen Tang, Chenghua Lin, Henglin Huang, Frank Guerin, and Zhihao Zhang. 2022. EtriCA: Event- triggered context-aware story generation augmented by cross attention. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5504-5518.
Hao Zhou, Tom Young, Minlie Huang. Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of the Eventt-Seventh Inter- national Joint Conference on Artificial Intelligence, IJCAI-18, International Joint Conferences on Artifi- cial Intelligence Organization, pages 4623-4629.
Chen Tang, Hongbo Zhang, Tyler Loakman, Chenghua Lin, and Frank Guerin. 2023. Enhancing dialogue generation via dynamic graph knowledge aggrega- tion. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 4604-4616.
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021a. Enhancing factual consistency of abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- Language Technologies, pages 718-733, Online. Association for Computational Linguistics.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2017. Graph attention networks. 6th International Conference on Learning Representations.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufci Li, Xiang Song, Jinjing Zhou, Chao Ma. Ling- fan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. 2019. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.011
Tianyu Zhu, Wen Hua, Jianfeng Qu, and Xiaofang Zhou. 2021b. Summarizing Long- Form document with rich discourse information. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, pages 2770- 2779, New York, NY, USA. Association for Comput- ing Machinery.

---

A Appendix
distantly relevant to the text for example, multi- word concepts with one matching word. Therefore, for each section of an article, we employed a sim- ple word overlap-based to filter out unwanted re- trieved concepts. Specifically, after removing stop- words from the main text and lemmatisation con- cept names, we retain only the concepts for which all words in its name were found to appear in the section text. We empirically found this approach to significantly outperform word embedding baaeedddnn--aedd approaches, which often failed filter out irrele- vant multi-word concepts a single word appeared in the main text.
Node type
Average Count
Document
1.00
Section 5.90
Metadata 7.49
Concept
364.93
SemType
63.47
Table 4: The average node type frequency statistics for a single article the train split.
Graph Statisties Table presents the average node type frequencies in the graph of given article. Additionally, Tables and 8 present the average semantic type (SemType) node frequencies, and Ta- ble 9 presents the average relation frequencies (all of which are located at the end of the Appendix).
Additional implementation and training details. We employ Longformer Encoder Decoder (LED) with the leeeeniille-bbseeee checkpoint as our base model for all knowledge enhancement approaches, using the default param- eters for model training and an input limit of 8192. For the "decoder cross-attention" method, replicate the standard Huggingface LED multi-head decoder cross-attention implementation (where the head no. determined by the model configuration), making use of the GAT-produced graph embedding as the key and value matrices (in place of the encoder output), with the query matrix being the output of the previous standard cross-attention module.
Huggingface
MetaMap UUMLS vocabularies. As mentioned in $3, we restrict MetaMap to a select num- ber of English vocabularies with access restric- tions lower than level 4. Specifically, we al- low MetaMap to use the following vocabularies: Alcohol and Other Drug Thesaurus (AOD), Dis- eases Database (DDB), CRISP Thesaurus (CSP), DrugBank (DRUGBANK), Diagnostic and Sta- tistical Manual of Mental Disorders Fifth Edi- tion (DSM-5), Human Phenotype Ontology (HPO), NCI Thesaurus (NCI), MedlinePlus Health Topics (MEDLINEPLUS) LOINC (LNC), MeSH (MSH), RxNorm (RXNORM), and Gene Ontology (GO).
For the "document embedding enhancement" method, we replicate the standard LED encoder layer, adapting the configuration to account for the larger input of the concatenated document and graph feature representations.
MetaMap Noise reduction. We found that MetaMap. in addition to accurately identifying UMLS concepts that were mentioned in pas- sage of text, often also returned large number of unwanted concepts. These typically were only
For the "article text augmentation" method, we prepend the article text with the graph-derived and shown in Figure extending the input text limit to 16,384 to accommodate the lengthy aug- mentation text without losing article text. All mod- els are trained for a maximum of 20 epochs, re- taining the checkpoints that achieved the highest validation set performance (average ROUGE score across variants).
IIIformation on all possible MetaMap vocabularies and their access restrictions can be found on the fol- lowing page: https:///wa. nlm, nih, gov/research/um sourceereleasedocs/inee html
Model
Abstractiveness (%)
The time taken to train each model on A100 GPUs ranged from 12 hours to 2 days depend- ing on the specific methodology used, with the GAT-based methods "'ocument embedding en- hancement" and "decoder cross attention" taking longer than the "article text augmentation" method.
Longformer text-aug
22.95
20.99
doc-enhance
20.00
decoder-attn
24.46
Table The average abstractiveness (measured in terms of novel I-grams) of summaries generated using differ- ent graph integration methodologies.
Summary abstractivenesss. In an effort to gain further insight into how each KG-enhancemenn

---

Relevance
Readability
Factuality
Model
R-11
R-21
R-L1
BeSt
CLI
DCRS
BaSt
Longformer
47.23
13.20
44.44
85.11
11.72
9.09
-2.56
text-aug + doc-enhance
46.28
13.45
43.77
84.72
11.37
8.06
-2.56
text-aug + decoder-attn
47.27
13.93
44.52
84.95
10.47
8.15
-2.57
doc-enhance + decoder-attn
28.85
3.7
27.58
73.60
2.68
1.56
-6.48
Table 6: Average performance of models with combinations of KG-enhancement methods on eLife test split. R ROUGE F1, BeS BERTScore F1, CLI Coleman -iau Index, BaS BARTScore.
method influences summary generation, we mea- sure the abstractiveness (as the percentage of novel 1-grams) of the generated summaries, providing the results in Table 5. Interestingly, the results show that both the "text augmentation" and "document enhancement" methods slightly decrease the ab- stractiveness of the base model, but the "decoder attention" method slightly increases it, suggesting that this method more effective at introducing external vocabulary.
Case study Figure 5 presents an expanded ver- sion of the case study given in Figure 4, whereby show all model outputs for each of the tech- nical concepts. As in Figure 4, KG-enhanced models typically improve on the ex- planation proved by the base Longformer model.
that the
Combining KG-enhancement Methods Table 6 presents the results obtained when applying dif- ferent combinations the KG-enhancement meth- ods to Longformer. Interestingly, no combined model matches the overall performance of any sin- gle KG method model. Although these combina- tions achieve positive results for readability metrics, this generally comes at the expense of readability metrics, with the combination of text augmenta- tion and decoder attention being the only model to achieve ROUGE scores equal to that of the base model.
We believe this performance degradation is likely result of the dilution or loss of document information at the expense of graph-based informa- tion. This is similar to what was observed when the value of p, given in equation (4), was set to too great a value for the document enhancement method. In the case of the method combinations that include the text augmentation method, the doc- ument information is likely more diluted as re- sult of the larger embedding size (as caused by the increased input size). Alternatively, when the doc- ument embedding and decoder attention methods are combined, the original document information seems be lost, resulting in particularly high loss values during training and the model being unable to produce a coherent output (as evidenced by the extremely low scores for readability and factuality metrics).

---

[Meiosis]
Longformer - During meiosis, the DNA in one of the chromosomes is copied and then the two copies are recom- bined so that each new generation will have single copy of the gene that encodes the protein encoded by that gene. 1/2
w/ text-aug process known as meiosis two copies of each chromosome are then exchanged between the newly formed cells, which results in unique of genes being passed to the next generation." 2/2
w/ doc-enhance meiosis depends on the rate and location of the recombination events, and on the length of the
DNA that is involved in the process. 1/2
w/ decoder-attn This is achieved through process called meiosis, which cell divides to produce four daughter cells, each of which contains a copy of one of the four copies of the genes found on the parent cell's chromosomes, 2/2
[Glabrous Skin Mechanoreceptors)
Longformer The orientation of an object depends largely how its edges activate mechanoreceptors in the
glabrous skin of the fingertips. 0/2
text-aug information about the orientation of object based largely on how its edges activate mechanore- ceptors the skin of fingertips, 2/2 tactile neurons the fingertips are able to detect the edges objects with high sensitivity2/22
w/ doc-enhance- The fingertip's 's surface is covered by a... layer of skin known as the glabrous skin.2/2 These cells are responsible for sensing touch, and they are also responsible for detecting the orientation of objects that touch them.? 2/2
w/ decoder-attn The orientation of object is determined largely by how its edges contact the skin and activate nerve cells in the fingertips.2. 2/2
[Slow Wave Sleep]
Longformer- Most studies of sleep have focused type of sleep known as slow wave sleep, in which the brain' activity alternates between periods of alternating periods of slow and fast sleep. 0/2
w/ text-aug slow wave sleep is thought to help us consolidate newly acquired memories and consolidate our of awareness of the world around us. 2/2 slow wave sleep, the propagation of slow waves is lost between the brain's 's regions.?!! 2/2
w/ doc-enhance Slow wave sleep is characterized by slow oscillation of electrical activity in the brain, 2/2 w/ decoder-attn Slow wave sleep is characterized by rhythmic waves electrical activity in the brain, which are thought to be part the process by which the brain consolidates memories," 2/2
Figure 5: extended case study comparing how the application of each method affects the explanation of specific technical concepts within the human evaluation sample. Colours and superscript are used to denote the number of evaluators who judged the sentence as readable for a lay audience (e.g.. 2/2 out of 2 evaluators).

---

/Concept definitions and relations)
Alleles Variant forms of the same gene, occupying the same locus on homologous CHROMOSOMES, and governing the variants in production of the same gene product. Alleles is a Gene or Genome.
Molecule = An aggregate of two or more atoms in a defined arrangement held together by chemical bonds.
Molecule a Substance.
Discover= = See for the first time; identify. Discover is a Activity.
Histocompatiiilitt- = The degree of antigenic similarity between the tissues of different individuals, which determines the acceptance rejection of allografts. Histocompatibility is Qualitative Concept.
In Vivo Located occurring in the body. In Vivo Spatial Concept.
Species A group of organisms that differ from all other groups of organisms and that are capable of breeding and producing fertile offspring. Species is a Classification.
Cells = The fundamental, structural, and functional units or subunits of living organisms. They are composed of CYTOPLASM containing various ORGANELLES and a CELL MEMBRANE boundary. Cells is Cell.
Allogeneic Taken from different individuals of the same species. Allogeneic is Qualitative Concept. Antigens Substances that are recognized by the immune system and induce immune reaction. Antigens is Immunologic Factor.
Result The result of action. Result is Functional Concept.
Major Histocompattbility Complex The genetic region which contains the loci of genes which determine the structure of the serologically defined (SD) and lymphocyte-defind (LD) TRANSPLANTATION ANTI- GENS, HUMAN; the IMMUNE RESPONSE GENES which control the ability of an animal to respond immunologically to antigenic stimuli, and genes which determine the structure and/or level of the first four components of complement. Major Histocompatibility Complex is Gene or Genome.
ANTIGENS, genes which control the structure of the IMMUNE RESPONSE-ASSOCIATEE
[SemTyye definitions]
Gene or Genome specific sequence, or in the case of the genome the complete sequence, of nucleotides along a molecule DNA or RNA (in the case of some viruses) which represent the functional units of heredity.
Substance = A material with definite or fairly definite chemical composition.
Activity An operation or series of operations that an organism or machine carries out or participates in. Classification = A term system of terms denoting an arrangement by class or category.
Cell The fundamental structural and functional unit of living organisms.
Qualitative Concept concept which is assessment of some quality, rather than direct measurement. Spatial Concept = A location, region, or space, generally having definite boundaries.
Immunologic Factor A biologically active substance whose activities affect or play a role in the functioning of the immune system.
Functional Concept A concept which is of interest because it pertains to the carrying out of a process or activity.
Quantitative Concept A concept which involves the dimensions, quantity or capacity of something using some unit of measure, which involves the quantitative comparison of entities. Temporal Concept A concept which pertains to time duration.
(Article texi)
Figure 6: Text augmentation example.

---

Semantic type
Average Count
Manufactured Object
0,98
Classification
0.61
Semantic type
Average Count
Qualitative Concept
1.0
Inorganic Chemical
0.66
or Syndrome
0.51
Pharmacologic Substance
0.93
Health Care Activity
0.95
Physiolugic Function
0.33
Occupational Activity
Immunologic Factor
0.58
Phenomenon or Process
0.99
Molecular Function
0.67
Individual Behavior
0.26
Clinical Attribute
0.39
Intellectual Product
1.0
Laboratory Procedure
0.88
Organism Attribute
0.96
Event
0.59
Health Care Related Organization
0.47
Human
0.04
Body Part, Organ, or Organ Component
0.86
Chemical Viewed Structurally
0.50
Social Behavior
0.67
Sign or Symptom
0.28
Therapeutic or Preventive Procedure
0.78
Enzyme
0.68
Research Device
0.13
Medical Device
0.78
Spatial Concept
1.0
Genetic Function
0.73
Temporal Concept
1.0
Nuclcic Acid, Nuclcoside, or Nuclcotide
0.75
Behavior
0.45
Organic Chemical
0.88
Environmental Effect of Humans
0.05
Patient or Disabled Group
0.23
Mammal
0.76
Virus
0.20
Research Activity
1.0
Cell Function
0.74
Quantitative Cuncept
1.0
Substance
0.94
Occupation or Disciplinc
0.81
Daily or Recrentional Activity
0.72
Functional Concept
1.0
Bacterium
0.37
Laburatory Test Result
0.17
Conceptual Entity
0.81
Neoplastic Process
0.17
Activity
1.0
Fukaryole
0.33
Population Group
0.65
Amino Acid Sequence
0.30
Age Group
0,35
Cell or Molocular Dysfunction
Mental Process
0,92
Regulation or Law Chemical
0.09
Natural Phenomenon or Process
0,93
Geographic Area
0,78
Body Space or Junction
Biologic Function
0.43
Nucleotide Sequence
Human- cansan Phenomenon Process
0.20
Chemical Viewed Functionally
Idea Concept
1.0
Family Group
Body Location Region
0.58
Vertebrate
Finding
1.0
Fungus
Organ Tissue Function
0.38
Amino Acid, Peptide, Protein
0.89
Embryonic Structure
Injury Poisoning
0.13
Professional Occupational Group
0.50
Molecular Biology Research Technique
Gene Genome
0.84
Molecular Sequence
Element, lon, Isotope
0,91
Group Attribute
Body Substance
0.56
Vitamin
Cell
0,91
Mental Behavioral Dysfunction
Receptor
0.50
Hormone
Hazardous Poisonous Substance
Plant
Pathologic Function
0.59
Anatomical Structure
0.06
Organism
0.46
Fish
0.14
Indicator, Reagent, or Diagnostic
0.81
Machine Activity
0.25
Cell Component
0.82
Educational Activity
0.09
Biologically Active Substance
0.91
Reptile
0.02
Animal
0.52
Amphibian
Biomedical or Dental Material
0.63
Congenital Abnormality
0.04
Group
0.38
Experimental Model of Disease
0.06
Body System
0.26
Archacon
0.02
Physical Object
0.34
Language
0.02
Antibiatic
0.39
Anatomical Abnormality
0.01
Organism Function
0.95
Acquired Abnurmality
0.01
Governmental or Regulatory Activity
0.48
Fully Formed Anatomical Structure
0.01
Organization
0.61
Professional Society
0.01
Tissue
0.40
Clinical Drug
0.00
Diagnostic Procedure
0.41
Self-help Relief Organization
0.00
Biomedical Occupation or Discipline
0.68
Entity
0.40
Table The average semantic node type frequency
statistics for single article in the train split (continued).
Table 7: The average semantic node type frequency statistics for single article in the train split.

---

Relation type
Average Count
contains
564,94
has keyword
1.59
has_title
4.90
was_ was_pubbissed__ig in
1.0
is_a
536.22
branch_of
0.86
affects
255.99
performs
25.23
exhibits
6.29
conceptual part_aa of
10.80
result of measures
134,99
71.28
issue in associated_ with
97.14
43.36
occurs_in connected_ to
15.44
2.61
process_of
101.00
degree_of
12.83
manifestation of
37.81
uses
14.00
location_oo of
60.24
causes
42.52
adjacent_to
6.13
tributary_oe of
0.86
prevents
6.84
produces
67.90
method_ee of property_oo _f complicates evaluation of
13.61
6.45
54.91
22.34
co-occurs_wind with carrics_out
21.40
8.23
interacts with measurement_cr of traverses
102.22
17.90
1.45
developmental_nne cactineetttaooomm_e_e__ form_of
3.52
treats
8.34
surrounds
4.74
part_ot of
27.68
conceptually_relatee recllyyclcldddooeeee
0.31
precedes
30.83
consists_of
6.54
analyzes
16.93
assesses ussesses_effect eefectt_ooo of
23.76
disrupts
49,62
contains
2.97
diagnoses
8.38
indicates
3.30
practices
0.78
manages
0.77
derivative_of _o
0.28
interconnects ingredient of
0.70
0.05
Table 9: The average relation type frequency statistics for a single article in the train split.


Reviews:
[{"review_id": "62PbYlwZPF", "reviewer": "Reviewer_sfG5", "summary": "", "questions": "Have you also applied also NER techniques to retrieve UMLS concepts from articles?\n\nHow long your approach need to train/test the model on your hardware? Please add these details.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and your recognition of our work. Please find our response to specific comments and questions below:\n\n__*The method is interesting, but it doesn't propose a very innovative approach, simply comparing three previous methods with slight modification to include the KGs.*__\n\nWhilst we respect the reviewer's perspective, we would like to highlight that none of these methods (or indeed any knowledge graph integration method) have previously been applied to the task of Lay Summarisation. In fact, as mentioned in the paper,  such methods have primarily only been applied to news article summarisation which is arguably a much less difficult task, given its significantly shorter and less abstractive reference summaries (as well as shorter input documents). Additionally, these methods have generally only been applied to internally derived knowledge graphs (i.e.,constructed from the source document itself). Therefore, combined with the various modifications made to each method and the application of these methods to external data, we would argue that the in-depth comparison of methods we offer for our specific task constitutes a significant innovation, and provides findings that will also be valuable to other types of long-form or abstractive summarisation/generation. \n\n__*Have you also applied also NER techniques to retrieve UMLS concepts from articles?*__\n\nIn preliminary experiments (not included in the paper), we did try using both general-purpose and domain-specific NER models to identify biomedical concepts mentioned in articles. However, we empirically found such models to obtain much lower recall on UMLS concepts than MetaMap, which was purpose-built for this exact task. Additionally, MetaMap offered the advantage of returning UMLS concept IDs (rather than just the entity name), which allowed for the easy retrieval of other information such as the semantic type and definitions of identified concepts. Therefore, we decided to use MetaMap with an additional layer of filtering (as discussed in section 3.1 of the paper), in order to maximise the amount of UMLS-derived information we could include in our KGs. We will make sure to include these details in the  final paper version.\n\n__*How long your approach need to train/test the model on your hardware? Please add these details.*__\n\nThe time taken to train each model on 2 A100 GPUs ranged from ~12 hours to 2 days depending on the specific methodology used (with the GAT-based methods - “document embedding enhancement” and “decoder cross attention” - taking  longer than the text augmentation method). We will make sure to add this information to the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, organized, structured and full of details.\nIt also applies the proposed method to biomedical lay summarization for the first time.", "reasons_to_reject": "The method is interesting, but it doesn't propose a very innovative approach, simply comparing three previous methods with slight modification to include the KGs.", "questions_to_authors": "Have you also applied also NER techniques to retrieve UMLS concepts from articles?\n\nHow long your approach need to train/test the model on your hardware? Please add these details.", "paper_topic_and_main_contributions": "The paper describes and compare three methods to improve the lay summary of biomedical scientific articles by exploiting article-specific knowledge graphs that contains information on relevant biomedical concepts described in the articles.\nThe article is well written, with examples of the adopted Knowledge Graphs, a well as the description of the tested summarization methods (Decoder cross-attention, Document embedding enhancement, Article text augmentation).\nFinally, the tests on eLife dataset have been evaluated quantitatively (using specific-summarization metrics) and qualitatively (by human readers).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jOORB6CeuO", "reviewer": "Reviewer_6F5o", "summary": "", "questions": "(A) Does the external knowledge graph decrease the abstractiveness of the summaries (target and generated)? In some sense adding the external vocabulary may be making the task more extractive than abstractive, which would be a contribution/finding worth including.\n\n(B) What was the global attention setting?\n\n(C) Why is it surprising that all methods of knowledge graph integration performed comparably? This is the same information, allowed to interact fairly richly with all the document tokens, and relatively minor differences in model architecture.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and your recognition of our work. Please find our response to specific comments and questions below:\n\n__*(arguably a presentation point) One of the paper’s main points is a lack of external knowledge in use while summarizing. LLMs are a source of external knowledge while summarizing. This is a major theme in the paper, and diminishes the work.*__\n\nWe are sorry, but we do not really understand this comment, since this paper does not use LLMs. We agree that the use of LLMs also  has the potential to introduce external knowledge for Lay Summarisation, and this is  something that we would definitely like to explore in future work. However, in this work we do not make use of LLMs, and there are still many scenarios where a the use of LLMs may not be possible or preferable (e.g., in lower-resource settings), thus making the integration of external knowledge into smaller LMs (such a Longformer) a topic  still worthy  of investigation.\n\n__*Graph Attention Networks are a single type of graph network. The paper documents different approaches to incorporating graph information into a transformer, not an exhaustive or even extensive search over graph representations, as claimed in the work.*__\n\nWe acknowledge that other types of graph network are also available and thank the reviewer for highlighting this. Our decision to make use of graph attention networks for our experiments was based on their popularity in the relevant literature (e.g., [Liu et al., 2021](https://ojs.aaai.org/index.php/AAAI/article/view/16796), [Huang et al. 2020](https://aclanthology.org/2020.acl-main.457/)) and, whilst we would also like to look into other types of graph representation in future work, it was beyond the scope of this work. We will make sure that this is made clear in the final version of the paper.  \n\n__*Manual evaluation was conducted over only 5 articles; while there may have been ~70 sentences evaluated, this does not give a particularly good measure for overall performance.*__\n\nWhilst we agree that a larger sample size would have been desirable, this was an unfortunate consequence of the costly nature of our human evaluation, which already demanded a significant cognitive effort and time commitment from our expert evaluators (primarily due to the fact each sentence had to be manually fact-checked against the input article). Again, we will endeavour to make this more clear in the final version. We would also like to highlight that the sample size is taken into account during the statistical significance testing carried out on the evaluation results, which demonstrate evidence of significant improvement (specifically, for readability) despite the relatively small sample size.\n\n__*Does the external knowledge graph decrease the abstractiveness of the summaries (target and generated)? In some sense adding the external vocabulary may be making the task more extractive than abstractive, which would be a contribution/finding worth including.*__\n\nWe thank the reviewer for raising this interesting point - to investigate it, we have rerun our evaluation with the addition of an “abstractiveness” measure (equal to the percentage of novel unigrams in the generated summaries) - the results for this are included below: \n\nLongformer (basline): 22.95%;\ntext-aug: 20.99%;\ndoc-enhance: 20.00%;\ndecoder-attn: 24.46%;\n\nInterestingly, the results show that both the ”text augmentation” and “document enhancement” methods do slightly decrease the abstractiveness of the base model (as suggested by the reviewer), but the “decoder attention” method actually slightly increases it. We will make sure to include this finding in the final version of the paper. \n\n\n__*What was the global attention setting?*__\n\nWe kept the global attention setting of Longformer to its default summarisation setting, as defined by the [HuggingFace](https://huggingface.co/docs/transformers/model_doc/led) (i.e., the first 
 token). Note that all code for our models will be made available in the final version. \n\n__*Why is it surprising that all methods of knowledge graph integration performed comparably? This is the same information, allowed to interact fairly richly with all the document tokens, and relatively minor differences in model architecture.*__\n\nIn the case of the text augmentation method, we felt that the performance similarity was interesting  as this method is actually exposed to significantly less graph information than the other two GAT-based methods (i.e., only the concept definitions and semantic types of the UMLS concepts mentioned in the article abstract  due to model input constraints). Although we agree that (for the reasons you have given) the performance similarity in the case of the GAT-based methods might not necessarily be surprising, we think that it is an interesting and important comparison to make due to the fact that each of these methods targets a distinct component of the model architecture for knowledge integration (i.e., the encoder and the decoder).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a well-implemented NLP engineering experiment for lay-summarization of scientific articles.", "reasons_to_reject": "(A) (arguably a presentation point) One of the paper’s main points is a lack of external knowledge in use while summarizing. LLMs _are_ a source of external knowledge while summarizing. This is a major theme in the paper, and diminishes the work.\n\n(B) Graph Attention Networks are a single type of graph network. The paper documents different approaches to incorporating graph information into a transformer, not an exhaustive or even extensive search over graph representations, as claimed in the work. \n\n(C) Manual evaluation was conducted over only 5 articles; while there may have been ~70 sentences evaluated, this does not give a particularly good measure for overall performance.", "questions_to_authors": "(A) Does the external knowledge graph decrease the abstractiveness of the summaries (target and generated)? In some sense adding the external vocabulary may be making the task more extractive than abstractive, which would be a contribution/finding worth including.\n\n(B) What was the global attention setting?\n\n(C) Why is it surprising that all methods of knowledge graph integration performed comparably? This is the same information, allowed to interact fairly richly with all the document tokens, and relatively minor differences in model architecture.", "paper_topic_and_main_contributions": "The primary contribution of this paper is the application of external knowledge graphs (derived from UMLS and MetaMap) in lay summarization. Secondarily, the paper investigates three methods of incorporating the external knowledge graph information into an encoder/decoder model: (A) by concatenating a graph encoding of the document and external knowledge in the decoder, (B) by concatenating the graph and document encodings and interpolating between this and an additional layer, and (C) linearizing the graph representation and passing this into the transformer. It includes a thorough automatic evaluation and a human evaluation over a small set of documents, demonstrating the approach improves performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Integrating graph based knowledge for summarization:\nhttps://link.springer.com/article/10.1007/s42979-023-01867-1 (this particular work considered contemporaneous, others not)\nhttps://www.mdpi.com/2079-9292/9/9/1520 \nhttps://arxiv.org/abs/2204.13761 \nAny of the retrieval augmented language models, e.g. https://arxiv.org/abs/2005.11401"}}, {"review_id": "y9vDCabEJ0", "reviewer": "Reviewer_cPxM", "summary": "", "questions": "There are no specific questions to the authors.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and your recognition of our work. Please find our response to specific comments below:\n\n__*The base model is LongFormer. It is possible that the conclusions do not carry to other base models.*__\n\nIn Longformer, we selected what was the most appropriate model for the task, given its general popularity and its ability to process the entire input document (that has resulted in strong performance on a number of similar tasks). However, we agree that exploring additional base models is a useful direction for future work.\n\n__*Only one dataset was used in the evaluation.*__\n\nAlthough we agree that it would have been desirable to include more datasets for evaluation, we opted to use eLife exclusively due to the particularly high quality of its reference lay summaries (as described in [Goldsack et al. (2022)](https://aclanthology.org/2022.emnlp-main.724/)). Specifically, eLife’s lay summaries are written by journal editors who are experts in communitcating complex technical findings to a lay audience - something which is unique amongst lay summarisation dataset, of which only a few exist.\n\n__*It is not stated whether the code will be available. This can make reproducibility more difficult, but this is compensated by the fact that the approach is explained in detail.*__\n\nWe can assure you that code will be made available and a GitHub URL included in a final paper version. This was simply not included in the current version so as to not break author anonymity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper is well written, the explanations are detailed.\n\nB. The three methods to inject knowledge graph information are intuitive and explore the impact of injecting such information in different stages of a base encoder-decoder architecture.\n\nC. The evaluation compares the approach against well known and strong baselines, and the results show an improvement.\n\nD. There is automatic evaluation using well-known evaluation metrics (ROUGE and BARTScore), plus an automatic assessment of readability. In addition, there is a human evaluation.", "reasons_to_reject": "A. The base model is LongFormer. It is possible that the conclusions do not carry to other base models.\n\nB. Only one dataset was used in the evaluation.\n\nC. It is not stated whether the code will be available. This can make reproducibility more difficult, but this is compensated by the fact that the approach is explained in detail.\n", "questions_to_authors": "There are no specific questions to the authors.", "paper_topic_and_main_contributions": "This paper presents an approach for lay summarisation of biomedical articles by incorporating information from knowledge graphs. The authors explore three different ways to inject the information from knowledge graphs in a base encoder-decoder architecture. The three approaches are independently tested and compared against state-of-the-art baselines, using the eLife dataset.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The related work section is comprehensive and I did not identify any important missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 573: \"This paper\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the use of knowledge graphs (KG) for improving lay summarization in the biomedical domain. Three strategies are proposed to integrate KG information into summarization models and experiments are conducted on the eLife dataset augmented with KG. Overall, reviewers are positive about this work. They found the paper well-written and acknowledge the novelty of applying KG integration to lay summarization. The authors  addressed all of the questions in the rebuttal, though there are remaining concerns about the size of the manual evaluation (only 5 articles).
Decision: Accept-Main