Biomedical Relation Extraction with Entity Type Markers and Relation-specific Question Answering
Year: 2023

Abstract:
Recently, several methods have tackled the relation extraction task with QA and have shown successful results. However, the effectiveness of existing methods in specific domains, such as the biomedical domain, is yet to be verified. When there are multiple entity pairs that share an entity in a sentence, a QA-based relation extraction model that outputs only one single answer to a given question may not extract desired relations. In addition, these methods employ QA models that are not tuned for relation extraction. To address these issues, we first extend and apply a span QA-based relation extraction method to the drug-protein relation extraction by creating question templates and incorporating entity type markers. We further propose a binary QA-based method that directly uses the entity information available in the relation extraction task. The experimental results on the DrugProt dataset show that our QA-based methods, especially the proposed binary QA method, are effective for drug-protein relation extraction. Our source code is available at https: //github.com/tticoin/BioRE-ETM-QA

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 377–384 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: In recent years, several methods that tackle the relation extraction (RE) task with question answering (QA) have shown impressive results (Levy et al., 2017; Li et al., 2019; Cohen et al., 2021). Instead of directly finding a relation between an entity pair in a sentence, in these methods, questions relating to the relation between the entity pair are answered by a QA model to determine whether the entity pair has the relation.
For example, suppose that we have the following sentence “Tom was born in 1999”. To determine whether the relation born_in exists or not between the entity pair Tom and 1999, Cohen et al. (2021) builds two questions, each containing one of the two entities, e.g., “When was Tom born?” for an entity Tom and “Who was born in 1999?” for an
entity 1999. If the answer from a span QA model is the other entity that is not in the question for one of these questions, the method determines that there is a relation “born_in” between the pair. Such a QA-based RE method can focus on each relation by preparing different questions for different relations.
However, existing methods have been applied only to general domain datasets, and their effectiveness has not been verified in specific domains, such as the biomedical domain. When there are multiple entity pairs that share an entity in a sentence, a QA model that can output only one answer to a question cannot extract relations since the questions become identical for the same entity. For example, when using a question “Who was born in 1999?” to determine if there is a relationship born_in between the entity pair (“Tom”, “1999”) and (“John”, “1999”) in the sentence “John was born in 1999, like Tom.,” the answer will be either Tom or John because the input to the model is the same. In this case, it is determined that there is no relation “born_in” for one of the entity pairs. Moreover, the employed QA models are not tuned for RE.
In this study, we extend and apply a span QAbased RE method (Cohen et al., 2021) for drugprotein RE to check the effectiveness of the method in the biomedical domain task. For this purpose, we manually create question templates for a drugprotein RE dataset DrugProt (Krallinger et al., 2021). To deal with each entity pair, even if multiple entity pairs share the same entity, we introduce entity type markers that incorporate entity types available in the RE task into the question and input sentence. Furthermore, we propose a binary QAbased method that directly classifies whether the answer to a question, which includes one entity of the target entity pair, is the other entity of the pair or not.
Our contributions are summarized as follows:
• We extend and apply a span QA-based RE method to the DrugProt drug-protein RE
377
dataset by creating question templates and incorporating entity type markers.
• We propose a binary QA-based method that directly uses the entity information available in the RE task.
• We show that the binary QA-based method is effective for drug-protein RE.

heading: 2 Related Work
text: Levy et al. (2017) have proposed the method of zero-shot RE using QA. Using an existing QA model, their method could make predictions even for unseen relation types by preparing questions corresponding to the relation types. Li et al. (2019) proposed the method for entity-relation extraction by multi-turn QA. The hierarchical structure of the relation types can be captured by extracting entities with QA at each turn and using them to construct questions at the next turn.
Cohen et al. (2021) proposed the RE method that uses QA with two-way questions, each containing a head or tail entity. First, they created two question templates for each relation type. One is a template in which the tail entity is answered using the head entity, and the other is a template in which the head entity is answered using the tail entity. If at least one of the answers to the two questions corresponds to the entity not used in the questions from the templates, it is determined that the entity pair has a relation type for which the template is created. They extract all the relation types by repeating this process for all the relation types.
QA-based methods have also been applied to other tasks, such as named entity recognition (Li et al., 2020b), event extraction (Li et al., 2020a; Liu et al., 2020), and coreference resolution (Wu et al., 2020). In the biomedical domain, the effectiveness of QA-based methods has been verified in event extraction (Wang et al., 2020) and spatial information extraction (Datta and Roberts, 2022).

heading: 3 Method
text: This section describes our RE models that employ the span QA-based RE following (Cohen et al., 2021) to drug-protein RE and proposes a binary QA-based method. We first introduce the question templates to apply QA-based RE to drug-protein RE in Section 3.1. We next introduce entity type markers, which incorporate the information about the entity types into the input sentence and question
in Section 3.2. We then explain a span QA-based method that predicts the start and end of the answer span in Section 3.3. We finally propose a binary QA-based method in Section 3.4.

heading: 3.1 Question templates
text: We created question templates for all relation types in the DrugProt dataset (Krallinger et al., 2021) to apply the RE method using QA to drug-protein RE. For each relation, we created two types of question templates: one is a template with a slot for the drug and the other for the protein, following the two-way QA format in (Cohen et al., 2021). DrugProt has 13 relation types, so the total number of question templates is 26. These question templates were created based on the annotation guidelines (Rabal et al., 2021) for the dataset. For example, the template for the relation “INDIRECT-DOWNREGULATOR” was set to “What does DRUG downregulate via other targets?” and “What downregulates PROTEIN via other targets?” using the part of the definition in the annotation guideline “CEM downregulates GPRO via other target” (CEM stands for drug and GPRO stands for protein). The templates are shown in Table 3 of Appendix A.

heading: 3.2 Entity type marker
text: We insert entity type markers into the input sentence and question to make the input for each entity pair unique and to use the entity types available in the RE task as input to the QA models. For each target entity pair, we assign the markers to the drug and protein entities in the input sentence and the question. The interrogatives are also assigned markers corresponding to the answer entity type to link the question and the answer.
Table 1 shows examples of inserting markers into input sentences and questions. First, the drugprotein pair (TT-B, kinase) and (TT-B, EGFR) are created from the entities in the input sentence “The kinase activity of EGFR was little inhibited by TTB in a cell-free system”. Next, drug markers (<D> and </D>) and protein markers (<P> and </P>) are assigned to the target entity pairs in the input and question sentences. In addition, the markers corresponding to the answer are assigned to the interrogative “What” in the question.

heading: 3.3 Span QA-based RE
text: To evaluate the RE method using QA, we build a span QA model that predicts the start and end indices of the answer span, as shown in Figure 1
following Cohen et al. (2021). First, entity type markers are assigned to the question and the input sentence, as shown in Section 3.2. Next, a sequence S = {w1, w2, · · ·wn}, which is the concatenation of the question and the input sentence with the [SEP] token between them, is input to Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) to obtain the representation of each token hi.
{h1,h2, . . . ,hn} = BERT(w1, w2, . . . wn) (1)
From the representation of each token, the probabilities psi and p e i that indicate whether the i-th token is the start and end tokens of the answer span are obtained by applying the fully connected layer and the softmax function.
zsi = W shi + b s (2) zei = W ehi + b e (3)
{ps1, ps2, · · · } = Softmax(zs1, zs2, . . . ) (4) {pe1, pe2, . . . } = Softmax(ze1, ze2, . . . ) (5)
If at least one answer span of two-way questions is the entity, the target entity pair is determined to have the relation type corresponding to the question. When the answer span is the entity for multiple relation types, the target entity pair is assumed to have multiple relation types.

heading: 3.4 Binary QA-based relation extraction
text: In the method of Section 3.3, the agreement between the answer span of the question and the entity span is checked. We consider this too much since it is sufficient to determine whether the answer is the target entity or not for RE. Therefore, we propose a binary QA-based method that uses a binary classification to determine whether the entity is the answer to the question based on the span representation of the entity that is not included in the question text, as shown in Figure 2.
In this method, the representations hi in Equation (1) of Section 3.3 are used to create the span representation of the candidate answer entity. The
span representation is created by concatenating the representations of the first and last tokens of the span and the mean of the representations of all tokens in the span, as shown below.
hmean = mean(hs, · · · ,he) (6) hspan = Concat(hs,h mean,he) (7)
Here, s and e denote the start and end positions of the entity span, and hspan denotes the span representation. The probability that the entity is the answer pspan is obtained by applying a fully connected layer and a sigmoid function to this span representation.
pspan = Sigmoid(Wspanhspan + bspan) (8)
Similarly to Section 3.3, classification is performed in two ways for all relation types, and an entity pair can have multiple relation types.

heading: 4 Experimental Settings

heading: 4.1 Dataset
text: We used the DrugProt drug-protein RE dataset for the evaluation. The dataset consists of abstracts from the pharmaceutical literature that contain drug and protein mentions and 13 types of drug-protein relations. DrugProt originally consisted of training, development, and test data, but the labels for the test data were not publicly available, so the documents in the original development set were split into 1:1 and used for our development and test data. Table 4 in Appendix B shows the statistics of the data. As a preprocessing, each document was split into sentences using SciSpacy (Neumann et al., 2019). We used the micro-averaged F-score for all relations for evaluation, which is calculated using evaluation scripts provided by the task organizer.

heading: 4.2 Compared methods
text: To verify the effectiveness of the QA-based methods, we compare the span QA-based method (SpanQA) in Section 3.3 and the binary QA-based method (BinQA) in Section 3.4. In order to evaluate the QA-based RE with the templates created
in Section 3.1, we experiment with the method using special tokens (STQA) specific to the relation types instead of the question templates to ensure that the natural language questions are effective or not and the method of RE by sentence classification using BERT’s CLS tokens (CLS). In STQA, for the relation “INHIBITOR”, instead of the natural language templates “What does DRUG decrease the activity of?” and “What decreases the activity of PROTEIN?”, the templates with special tokens “<INHIBITOR> DRUG?” and “</INHIBITOR> PROTEIN?” are used. Furthermore, to confirm the effectiveness of the entity type markers, we evaluated BinQA without the markers.
All methods used PubMedBERT-base-uncasedabstract-fulltext (Gu et al., 2022), which has been pre-trained using a large biomedical literature, as an encoder. Adam (Kingma and Ba, 2015) was used as the optimization method, and the learning rate was set to 3e-6. The threshold for the binary QA-based method described in Section 3.4 was set to 0.7. In addition, we trained and evaluated the methods five times with different seeds and calculated the average score.

heading: 5 Results
text: Table 2 shows the micro-averaged F-scores for the span QA-based methods (SpanQA), the proposed
method (BinQA), the method using special tokens (STQA) instead of question templates, and the RE method using sentence classification (CLS)1. BinQa improved the micro-averaged F-score by 1.6 percentage points on the test set compared to the SpanQA. These results indicate that the binary QAbased method is effective for RE. In a comparison of BinQA and STQA, the proposed method has a 1.0 percentage point higher micro-averaged F-score than STQA on the test set, indicating the effectiveness of using natural language for the questions. In addition, a comparison between the QA-based methods (SpanQA, BinQA, and STQA) and the sentence classification method (CLS) shows that the QA-based methods are also effective in drug-
1We summarize the RE performance for each relation type in Appendix C
protein RE. When the markers were removed from BinQA, the micro-averaged F-score decreased by 2.1 percentage points on the test set. These results show that the entity type markers are effective.

heading: 6 Conclusions
text: In this paper, we applied and extended span QAbased RE method to the drug-protein RE by creating question templates and incorporating entity type markers. We also proposed the binary QAbased method that directly uses the entity information available in the RE task. We showed the QA-based methods, especially the proposed binary QA method, are effective for drug-protein RE. For future work, we will analyze the advantages and problems of the proposed QA-based methods by comparing them with other RE methods in the biomedical domain. In addition, we will analyze difficulties in the biomedical domain by applying and comparing QA-based methods in the general domain.
Limitations
In this study, we used entity type markers to enable a QA model to output multiple answers for RE for multiple entity pairs that share an entity in a sentence. However, we have not evaluated other QA models that can output multiple answers. In addition, since entity type markers are incorporated before and after entities, it is difficult to apply the method to nested entity pairs.
QA-based RE methods are not computationally efficient since they require multiple QA processes to extract the relation for an entity pair. Furthermore, the question templates in QA-based RE affect the performance of RE, but in this study, only one template set was used for evaluation, and there is room for improvement.
Direct comparison with existing SotA (Weber et al., 2022) is not performed because of the unavailability of the original DrugProt test data set. In addition, the SotA model is an ensemble of 10 pretrained language models, and we cannot directly compare the performance in a fair setting.

heading: Acknowledgements
text: This work was supported by JSPS Grant-in-Aid for Scientific Research JP20K11962.

heading: A Question templates
text: Table A shows the question templates for the DrugProt dataset. Two templates are defined for each relation type, and each template consists of a question with a slot for a drug or a protein. Before inputting the model, the DRUG or PROTEIN of the template is replaced with the target entity.

heading: B Dataset statistics
text: Table 4 shows dataset statistics for the DrugProt dataset. As explained in Section 4.1, the documents in the original development data were split and used for our development and test data. The training data consist of 3,500 documents, and the development and test data consist of 375 documents each.

heading: C RE performance for each relation type
text: Tables 5 and 6 show the F-scores for each relation type and micro-averaged F-scores for the methods described in Section 4.2. We trained and evaluated the methods five times with different seeds and calculated the average scores.

heading: Future Work
text: For future work, we will analyze the advantages and problems of the proposed QA-based methods by comparing them with other RE methods in the biomedical domain. In addition, we will analyze difficulties in the biomedical domain by applying and comparing QA-based methods in the general domain. Limitations
In this study, we used entity type markers to enable a QA model to output multiple answers for RE for multiple entity pairs that share an entity in a sentence. However, we have not evaluated other QA models that can output multiple answers. In addition, since entity type markers are incorporated before and after entities, it is difficult to apply the method to nested entity pairs. QA-based RE methods are not computationally efficient since they require multiple QA processes to extract the relation for an entity pair. Furthermore, the question templates in QA-based RE affect the performance of RE, but in this study, only one template set was used for evaluation, and there is room for improvement. Direct comparison with existing SotA (Weber et al. , 2022) is not performed because of the unavailability of the original DrugProt test data set. In addition, the SotA model is an ensemble of 10 pretrained language models, and we cannot directly compare the performance in a fair setting