How Much do Knowledge Graphs Impact Transformer Models for Extracting Biomedical Events?
Year: 2023

Abstract:
Biomedical event extraction can be divided into three main subtasks; (1) biomedical event trigger detection, (2) biomedical argument identification and (3) event construction. This work focuses in the two first subtasks. For the first subtask we analyze a set of transformer language models that are commonly used in the biomedical domain to evaluate and compare their capacity for event trigger detection. We fine-tune the models using seven manually annotated corpora to assess their performance in different biomedical subdomains. SciBERT emerged as the highest performing model, presenting a slight improvement compared to baseline models. Then, for the second subtask we construct a knowledge graph (KG) from the biomedical corpora and integrate its KG embeddings to SciBERT to enrich its semantic information. We demonstrate that adding the KG embeddings to the model improves the argument identification performance by around 20 %, and by around 15 % compared to two baseline models. Our results suggest that finetuning a transformer model that is pretrained from scratch with biomedical and general data allows to detect event triggers and identify arguments covering different biomedical subdomains, and therefore improving its generalization. Furthermore, the integration of KG embeddings into the model can significantly improve the performance of biomedical event argument identification, outperforming the results of baseline models.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 145–155 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Biomedical event extraction is a complex information extraction task that identifies key information from large sets of textual data for further applications, such as the study of biomolecular mechanisms or epigenetic changes. A biomedical event is constructed from an event trigger and one or more arguments that orbit around the trigger. Event triggers generally refer to nouns or verbs that express an action, circumstance or eventuality, while
the arguments refer either to biomedical entities or to other events, called nested events. Figure 1 1 shows the example of a sentence containing two biomedical events, ‘-Reg’ (which stands for ‘Negative regulation’) and ‘Locl’ (which stands for ‘Localization’). The event constructed from the trigger word ‘excretion’ of type ‘Locl’ (the event is given the same type as the trigger) presents as single argument the biomedical entity of type ‘D/C’ (which stands for ‘Drug or compound’), playing the role of ‘Th’ (which stands for ‘Theme’). This role allows answering the question ‘What is excreted?’. On the other hand, the event constructed from the trigger word ‘reduces’ of the type ‘-Reg’, presents two arguments. The first argument is a biomedical entity of the type ‘Drug or compound’, playing the role of ‘Cause’. This role allows answering the question ‘What causes the reduction?’. The second argument is the nested event ‘Locl’ described before, playing the role of ‘Theme’, answering the question ‘What is reduced?’.
Event extraction is usually divided into three main subtasks, event trigger detection, argument identification and event construction. Event trigger detection identifies and classifies the trigger words into a set of predefined types of event triggers, while argument identification identifies and classifies the roles between the event triggers and their respective arguments (Shen et al., 2019). Event construction refers to the unmerging of the arguments that correspond to the same event for its
1The visualization of the annotated sentence is done using the visualization tool brat https://brat.nlplab.org/.
145
construction (Björne and Salakoski, 2011).
Event trigger detection has a fundamental role in the construction of events. Indeed, the triggers are the targets that allow us to know that an event may exist (Cui et al., 2020). This subtask is usually considered as a classification problem, where each word need to be classified into a predefined set of trigger types. Difficulty for trigger detection comes from the sensitivity to the domain or subdomain (text can present specialized language), linguistic forms (triggers can be single words, multi-words, discontinuous markers) and ambiguity on the trigger class (a trigger word can be given different trigger classes) (Zerva and Ananiadou, 2015).
Argument identification can be also considered as a multi-category classification problem, where the directed relation between a trigger and an entity or other event needs to be classified into a predefined set of role types. When these arguments are correctly identified, the event extracted has the potential to provide a reliable means of improving domain knowledge. One of the main complexities in identifying arguments is that they can be part of one or multiple events (one-to-one and one-tomultiple relations), where they play the same or different roles.
Following (Ramponi et al., 2020), event trigger detection is the main source of errors in event extraction, where around 31 % of the errors correspond to non-detection of triggers and 28 % to overdetection of triggers. Further, the non-detection of arguments represents around 23 % of errors and the over-detection of arguments around 7 %. Transformer language models have been widely adopted to try to reduce errors in event extraction due to their positive achievements in performance for solving different types of Natural Language Processing tasks. BERT (Devlin et al., 2018), which stands for Bidirectional Encoder Representations from Transformers, is a language model designed to pretrain bidirectional representations of words, taking into account the semantics by considering both left and right directions of the text. From this pretraining, BERT can be fine-tuned by including additional layers on top of the model to solve new specific tasks. Furthermore, a number of domain-specific BERT variants have been developed by being trained on large corpora with the same context, such as the biomedical domain. However, since the learning of the models is limited to the subdomain in which they were trained, they present limitations in per-
formance when using them in different biomedical subdomains
To improve the integration of domain knowledge, knowledge graph (KG) models have been implemented along with language models for different information tasks in the biomedical domain (Huang et al., 2020; Yang et al., 2020; Dasgupta et al., 2021; Roy and Pan, 2021; Milošević and Thielemann, 2023). Biomedical KGs are a resource of integration of one or more sources of information (often manually curated datasets) into a graph, where biomedical entities can be represented by nodes and the relations between them by edges (Nicholson and Greene, 2020). KG models integrate nodes and edges into a low-dimensional vector space, known as embeddings, preserving the semantic information of the KG.
In this work, we first analyze the performance of five previously trained transformer language models to identify whether they allow the identification of triggers in different biomedical subdomains. Then, we enrich the semantic information of the best-performing model with KG embeddings to assess whether integrating these embeddings improves the model’s ability to identify biomedical arguments and their roles. For this purpose, BERT, BioBERT, SciBERT, PubMedBERT, and BioMedRoBERTa are fine-tuned using two different classifiers, a linear layer and a Bidirectional Long Short Term Memory (Bi-LSTM) layer, to detect biomedical event triggers. These BERT variants are chosen for comparison since they share the same BERT architecture but have previously been pretrained with different data in the biomedical and/or general domain, showing positive results in biomedical information extraction tasks (Lee et al., 2020; Beltagy et al., 2019; Erdengasileng et al., 2022). Models are learned using seven manually annotated data sets merged together. These corpora were originally developed for the event extraction task in different biomedical subdomains. Then, a KG is constructed from the biomedical events contained in the biomedical corpora and its KG embeddings are computed. These embeddings are integrated into the transformer language model to classify the roles between the previously identified triggers and the biomedical entities and/or other triggers, in order to detect the event arguments.

heading: 1.1 Contributions
text: Our main contributions include (1) evaluation and comparison of five transformer language models based on BERT for the detection of biomedical event triggers, (2) proposal of a novel strategy to integrate KG embeddings into transformer language models to identify biomedical event arguments, (3) empirical analysis of the effectiveness of merging annotated corpora to detect biomedical event triggers and identify arguments on different biomedical subdomains.

heading: 2 Related Work
text: (Rahul et al., 2017) use Recurrent Neural Networks (RNN) to extract higher level features through the hidden state of the network to identify biomedical event triggers. They also use the word and the entity type embeddings as features, demonstrating positive results in the MLEE (Pyysalo et al., 2012) corpus. (Duan et al., 2017) and (Zhao et al., 2018) explore an augmentation of the semantic information by integrating the full document representation. Both propose the use of RNNs to extract cross-sentence features without the use of external resources. (Nguyen and Grishman, 2018) present a Graph Convolution Network (GCN) model to exploit syntactic dependency relations. They use dependency trees to link words to their informative context for event trigger detection. (Yan et al., 2019) also propose a GCN model, integrating aggregative attention to model and aggregate multiorder syntactic representations of the sentences, while in the case of (Cui et al., 2020), they extend the GCN by adding the relation aware concept, which exploits the syntactic relation labels and models the relation between words. DeepEventMine (Trieu et al., 2020) is an end-to-end system for event extraction that consists in four main modules that identify the event triggers and the arguments. For each of the modules, BERT is used as base model and a linear layer is added. It has achieved the SOTA performance on seven biomedical nested event extraction tasks. (Portelli et al., 2021) compare BERT and five of its variants for the identification of Adverse Drugs and Events (ADEs). They show that span-based pretraining from spanBERT provides an improvement in the recognition of ADEs. Besides, the pretraining of the models in the specific domain is useful in comparison to train the models from scratch. (Ramponi et al., 2020) developed BEESL, a neural network
model based on a sequence labeling system for the extraction of events. The system converts the event structures into a format of sequence labeling, and uses BERT as language model. (Chen, 2021) propose the Multi-Source Transfer Learningbased Trigger Recognizer system, which is an extension on transfer learning using multiple source domains. Datasets from different domains are used for jointly train the neural network, achieving a higher recognition performance on the biomedical domain, having a wide coverage of events.
KG models have recently been also used for information extraction tasks in the biomedical domain. (Sastre et al., 2020) proposes a model based on a Bi-LSTM to extract drug information from drug labels and integrate it into knowledge graphbased embedding space to evaluate drug label accuracy. In (Huang et al., 2020) is proposed to detect relations between entities in biomedical events using a question answering approach. They incorporate domain knowledge into a pretrained language model using Graph Edge-Conditioned Attention Networks (GEANet), showing improved capabilities in inferring complex events. (Lai et al., 2021) presents a GCN network with attention for biomedical entity and relation extraction based on knowledge graphs embeddings. They first construct a KG by predicting the links between the biomedical entities and then make the predictions by merging the word entities and the embeddings. (Fei et al., 2021) proposes BioKGLM, a system where a pretrained language model is enriched by integrating large biomedical knowledge graphs. To effectively encode knowledge, they explore different fusion strategies to facilitate knowledge injection.
According to these works, transformer architectures have achieved competitive performance for extracting biomedical events, and the use of pretrained language models has shown an improvement in the performance of this task. However, nondetection, over-detection and misclassification of triggers continues being the most important cause of errors in event extraction (Ramponi et al., 2020). Besides, most of these works have been developed in a specific biomedical subdomain, not allowing a generalization to different subdomains. This is a limitation in the extraction of biomedical events because the biomedical language in texts is usually specialized and very specific.
We present an alternative approach to overpass this limitation, combining corpora from different
biomedical subdomains to train transformer language models in a broader biomedical domain. Besides, we enrich the context of the domain-specific language model SciBERT using KG embeddings. This strategy adds semantic knowledge about complex links that allow inferences between biomedical concepts that are not directly related. Our proposal outperforms significantly two strong baselines identifying arguments in the biomedical Cancer Genetics corpora.

heading: 3 Method
text: This work proposes the approach shown in Figure 2. The annotated data is given as input to the pretrained transformer language models for finetuning and then passed to a classification layer for event trigger detection. At this stage, a Named Entity Recognition (NER) task is performed to identify and classify the triggers in the text. Simultaneously, a KG is built from the annotated data and the KG embeddings are calculated. These embeddings are later integrated to the transformer language model and the fine-tuning is done to identify arguments. Here, a Relation Extraction (RE) task is performed to identify and classify the roles between the triggers and candidate arguments.

heading: 3.1 Transformer Language Models: BERT
text: BERT (Devlin et al., 2018) is the first and the basis of transformer language models. It is a contextualized word representation model based on a masked language model pretrained with bidirectional transformers. In BERT, the sequence of input tokens (words or sub-words) is constituted with initial vectors that are the combination of the token embeddings, the (token) position embeddings and the segment embeddings (text segment to which the token corresponds) through element-wise summation. The embeddings are then passed to a set of layers of transformer modules. Each transformer layer generates a contextual representation of every token by summing the non-linear transformation of the tokens’ representations from the previous layer. This representation is weighted by the attentions calculated using the representations of the previous layer as query. The last layer generates the contextual representations for all the tokens, where the information of the whole text span is combined.
Following the BERT principle, other transformer models have been pretrained with data from specific domains, e.g. biomedical data, present-
ing better adaptation for solving in-domain tasks. BioBERT (Lee et al., 2020) and BioMedRoBERTa (Gururangan et al., 2020) are some examples of BERT variants pretrained in the biomedical domain.

heading: 3.2 Biomedical Trigger Detection using Transformer Language Models
text: Various downstream text mining tasks can be performed by making minimal modifications to the BERT architecture, through a process of finetuning. Here, the transformer models are fine-tuned for NER, which aims to recognize domain-specific nouns in a corpus by giving each word in a sentence a predefined class. Since here NER is adapted to detect triggers, it implies not only identifying nouns, but also verbs and in some cases adjectives. For this purpose, after obtaining the contextual representation of the tokens in vectors, a classification layer is added to classify these vectors into the event trigger classes. Two different classification layers are used separately for comparison, a linear layer and a Bi-LSTM layer. The output labels are obtained following the IOB (Inside-OutsideBeginning) tagging to classify the triggers into the predefined trigger categories (in the case of the I and B tags).

heading: 3.3 Biomedical Argument Identification using Transformer Language Models and KG Embeddings
text: Biomedical event argument identification refers to finding the arguments that belong to an event and the role they play in it. A strategy to identify the arguments is RE, where the goal is to capture and classify the relations between triggers and biomedical entities and/or other triggers. Here, RE is applied by enriching with the KG embeddings of the triggers, the roles and the arguments the semantic information of transformer language model.
For this, a KG is first constructed from the gold data, where the semantic types of the triggers and biomedical entities are considered as the nodes, and relations between them are considered as the directed edges from the triggers to the arguments. These relations represent the role that the argument is playing with respect to the trigger. A KG is a knowledge base presented in a graphical structure format, composed by multiple types of relations between entities. KGs can be represented as a set of facts in the form of triples (h, r, t), where h, r and t represent the head, relation and tail, respectively. In
order to model and infer the relations in the triples, KG embeddings are calculated, mapping the content of entities and relations to low-dimensional continuous vectors to be later used for predicting new relations (Wang et al., 2021). A scoring function is used to measure the reliability of the triples based on the embeddings, where a higher score means that the triple is more likely to be true.
KG models are evaluated by ranking all the triples according to their scores and calculating the standard evaluation metrics Mean Reciprocal Rank (MRR) and Hits@z, where z ∈ 1, 3, 10 (Kristiadi et al., 2019; Islam et al., 2021). Both metrics scores range [0,1], where the higher value demonstrates the better ranking of positive test triples, which means a better prediction performance. Here, the KG embeddings were calculated using SimplE, a model based on the tensor factorization approach, canonical polyadic decomposition (Sorber et al., 2013). It learns independent embedding vectors for the entities in the head and the tail, even if they are tied. SimplE encodes the embeddings of the two entities h and t into h⃗ and t⃗, respectively, by parameter sharing allowing to integrate the dependence between them into a relation vector v⃗r and v⃗r−1 for the inverse relation. These embeddings are optimized by satisfying the scoring function of Equation 1.
1 2 (⟨⃗h, v⃗r, t⃗⟩+ ⟨⃗t, v⃗r−1, h⃗⟩) (1)
Finally, the KG embeddings are integrated to the embeddings of the transformer language model and then passed to a linear classification layer to identify the roles.

heading: 3.4 Incorporating KG Embeddings to Transformer Language Models
text: To obtain the KG embeddings, the total triples represented as (trigger type, role, argument type), are randomly split into 80 % to train and 20 % to test the KG model and obtain the embeddings. These embeddings are incorporated to the contextual representation of the transformer language model following the three different strategies described below. vtr and var represent the embeddings of the trigger and the argument from the transformer language model, respectively, and kgtr, kgar, kgr represent the KG embeddings of the trigger, argument and role, respectively.
KGtr,ar = [vtr; kgtr; var; kgar] (2)
KGr = [vtr; var; kgr] (3)
KGtr,r,ar = [vtr; kgtr; var; kgar; kgr] (4)

heading: 4 Experimental Settings

heading: 4.1 Corpora
text: Table 1 presents the seven publicly available datasets used for fine-tuning the transformer models. These data were manually or semi-manually annotated by experts and released to be used in the development and improvement of event extraction models.
Cancer Genetics (CG) 2013 (Nédellec et al., 2013) contains information of bio-processes in the development and progression of cancer. Epigenetics and Post-translational Modifications (EPI) 2011 (Ohta et al., 2011) focuses on proteins and DNA modifications. GENIA 2011 (Kim et al., 2011) and
GENIA 2013 (Kim et al., 2013) present both information about the transcription factors in blood cells, but this last updated with more recent articles. Infectious Diseases (ID) 2011 (Pyysalo et al., 2011) consists of data about biomolecular mechanisms of infectious diseases, virulence and resistance. Pathway Curation (PC) 2013 (Nédellec et al., 2013) focuses on targets reactions relevant to the development of biomolecular pathway models. MultiLevel Event Extraction (MLEE) (Pyysalo et al., 2012) presents different levels of biological organization ranging from the subcellular to the organism level.
For the development of the experiments, the training and development datasets of all the corpora are initially merged into one single dataset and split into sentences, obtaining a total of 24,819 sentences. The sentences are then split (following the split of the KG training and test sets) to have a set of 80 % to train and 20 % to test, containing 19,855 and 4,964 sentences, respectively. We do not use the original test datasets for the experiments since the annotations are not released. All the trigger types from each dataset are considered for the final trigger classification, presenting a final set of 58 trigger types (some types are overlap among the different corpora).

heading: 4.2 Knowledge Graph Construction
text: The KG is constructed from the events contained at the document level from the total dataset. The nodes represent the biomedical entities and the triggers, and the edges, which are directed from the triggers to the arguments, represent the roles. Nodes contain the information about the semantic type of the biomedical entities or the triggers, while edges contain the information about the role type (e.g. (Locl, Th, D/C), from the event Locl in Fig. 1). All the types of triggers, biomedical entities and roles from each dataset are considered for the
KG construction, presenting a final set of 81 types of triggers and biomedical entities and 12 types of roles (some types overlap among the different corpora).
From this step, 3,387 graphs are constructed. Then, the graphs are post-processed, keeping only those that present at least two nodes and one edge, reducing the total number of graphs to 2,721. All KGs are finally merged into one single graph through a disjoint union, containing 99,251 nodes and 56,931 edges.

heading: 4.3 Transformer Language Models
text: We compare BERT (Devlin et al., 2018), and four BERT variants pretrained in the biomedical domain, BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al., 2020), and BioMedRoBERTa (Gururangan et al., 2020) for the detection of event triggers. The models differ from each other by the corpora in which they were pretrained (all in English), their type of pretraining and the size of their vocabulary, as shown in Table 2.
SciBERT and PubMedBERT were pretrained from scratch using an unique vocabulary and include embeddings that are specific for in-domain words. BioBERT and BioMedRoBERTa were pretrained starting from the BERT checkpoints, their vocabularies are built with general-domain texts (similar to BERT) as well as the initialization of the embeddings.

heading: 4.4 Parameters Settings
text: Experiments are developed using NVIDIA GeForce GTX 1080 Ti (11 GiB) GPU and GeForce RTX 2080 Ti (11 GiB) GPU. The fine-tuning of the transformer models are done with PyTorch, using the Transformers package and the models are taken from Hugging Face 2. The transformer
2https://huggingface.co/
models are fine-tuned using the original parameters from BERT, leaving the last layer unfrozen.The training parameters of the classification layers, both linear and Bi-LSTM, are set as follows; batch size of training and test sets of 16, learning rate of 1e-05 and max gradient norm of 10. The maximum length of the sentences is set to 256. All the models are trained during 10 epochs on the training set and evaluated on the test set. The KG is constructed with NetworkX (Hagberg et al., 2008) and the model used to calculate the KG embeddings is implemented with PyTorch, using the base code from (Islam et al., 2021). It is trained using Adam optimizer, with a learning rate of 1e−4, a weight decay value of 0.01, pair as loss function and 4 as margin value. The training is conducted during 500 epochs, setting 768 as hidden size (embedding dimension) to match with the hidden size of the transformer language model.

heading: 5 Results and Discussion
text: We present a comparison of five different transformer language models for the detection of biomedical trigger events using seven biomedical corpora together. Also, we conduct different strategies for the enrichment of a transformer language model using KG embeddings for the identification of biomedical arguments. We finally compare our results using the test set of the CG dataset with two baselines models from the state-of-the-art.
Table 3 compares the five transformer language models used for the detection of event triggers. For each model is shown the result of adding a linear and a Bi-LSTM classifier. We observe that almost all the models present a better performance when a Bi-LSTM classifier is used, with the exception of BERT, which presents an F1-score of about 2 % higher when using a linear classifier. The highest results are presented in bold, corresponding to SciBERT-Bi-LSTM, a model pretrained from scratch using biomedical and general data. PubMedBERT, a model pretrained from scratch using
biomedical data, achieves the second best performance when Bi-LSTM is used as classifier, being below SciBERT by around 5 %.
These last three models, SciBERT, PubMedBERT and BERT, present similar characteristics. They are all pretrained from scratch using very comparable text sizes and have similar vocabulary sizes. However, BERT is pretrained only in the general domain, PubMedBERT in the biomedical domain, and SciBERT in both the general and biomedical domains. The two models that present the lowest performance are BioBERT and BioMedRoBERTa. Both models are pretrained from the BERT weights using biomedical and, biomedical and general data, respectively, and present the largest text sizes of all the models. There is an improvement in both models of around 7 % using a Bi-LSTM classifier compared to a linear classifier. However, their F1-score is around 14 % lower than the one obtained by SciBERT-Bi-LSTM.
Table 4 compares our model trained only on the CG corpus with two baseline event extraction models, TEES-CNN (Björne and Salakoski, 2018) and DeepEventMine (Trieu et al., 2020). TEES-CNN is a pipeline model based on a CNN architecture
for event extraction that sequentially applies event trigger detection, argument identification and the construction of events. DeepEventMine is a joint model based on BERT for event extraction that simultaneously detect event triggers, identify arguments and construct the final events. Both models presented state-of-the-art results in the CG task. The results reveal that our proposal achieves better F1-score than TEES-CNN by around 3 % and by DeepEventMine by around 1 %.
After observing that SciBERT presents the best performance in trigger detection, we enrich its semantic information with the KG of the biomedical corpora for argument identification. We first obtain the KG embeddings of the KG constructed from the seven biomedical corpora. Table 5 compares five KG models used for the computation of these embeddings, TransE (Bordes et al., 2013), TransH (Wang et al., 2014), TransD (Ji et al., 2015), DistMult (Yang et al., 2014), SimplE (Kazemi and Poole, 2018). The highest results are presented in bold, obtained with SimplE, which is the model that we use to compute the KG embeddings that are integrated to SciBERT.
Table 6 compares the results of the different strategies followed to integrate the KG embeddings to SciBERT. SciBERT-KGtr,r,ar, which integrates the KG embeddings of the trigger, the role and the argument, presents the highest F1-score. This strategy improves the F1-score by around 18 % compared to when KG embeddings are not inte-
grated. However, we observe that when we add the KG embeddings of the trigger and the argument in SciBERT-KGtr,ar, the performance improves by only 1 %, while when we add the KG embeddings of the role in SciBERT-KGr, the improvement is of 17 %. This reveals that integrating the KG embeddings to SciBERT allows to improve the performance in the identification of biomedical arguments, especially when the KG embeddings of the role are used.
Table 7 compares our model with TEES-CNN and DeepEventMine for the identification of arguments on the CG corpus. The results reveal that our proposal achieves a better F1 score than TEESCNN by around 16% and than DeepEventMine by around 15%, which could be significant in reducing event extraction errors related to argument identification.

heading: 6 Conclusions
text: We analyze different transformer language models for biomedical event trigger detection and argument identification using seven different corpora and KG integration. By comparing the performance of the models, we found that fine-tuning SciBERT with a Bi-LSTM classifier is the best strategy to detect events in different biomedical domains. Furthermore, when SciBERT is enriched with KG embeddings, especially those corresponding to the roles of arguments in events, it significantly improves the identification of biomedical arguments.
Limitations
Our work follows a pipeline approach, where the optimization of event trigger detection and argument identification are done separately. This causes the event trigger detection errors to be passed to the argument identification step. In the future we plan to do the joint optimization of both steps to reduce internally transmitted errors.
Ethics Statement
This work complies with the rules expressed in the ACM Code of Ethics. The NLP application in this work refers to information extraction in the biomedical domain. All the NLP tools and datasets used are mentioned and cited. We do not present a new dataset. The datasets utilized are used as intended. We do not use demographic or identity characteristics information. Our experiments involve around 500 hours of compute time.

heading: Future Work
text: In the future we plan to do the joint optimization of both steps to reduce internally transmitted errors