Distantly Supervised Document-Level Biomedical Relation Extraction with Neighborhood Knowledge Graphs
Year: 2023

Abstract:
We propose a novel distantly supervised document-level biomedical relation extraction model that uses partial knowledge graphs that include the graph neighborhood of the entities appearing in each input document. Most conventional distantly supervised relation extraction methods use only the entity relations automatically annotated by using knowledge base entries. They do not fully utilize the rich information in the knowledge base, such as entities other than the target entities and the network of heterogeneous entities defined in the knowledge base. To address this issue, our model integrates the representations of the entities acquired from the neighborhood knowledge graphs with the representations of the input document. We conducted experiments on the ChemDisGene dataset using Comparative Toxicogenomics Database (CTD) for documentlevel relation extraction with respect to interactions between drugs, diseases, and genes. Experimental results confirmed the performance improvement by integrating entities and their neighborhood biochemical information from the knowledge base. 1

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 363–368 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The number of documents reporting new interactions and functional/pathway relationships between biochemical entities continues to increase rapidly, and manual registration and maintenance of knowledge bases are becoming increasingly difficult to keep up with the pace (Davis et al., 2021; Wishart et al., 2017). Automatic extraction of biochemical relationships from documents could help in maintaining. Machine learning approaches have been the mainstream of relation extraction (RE) for more than a decade due to their high performance. However, machine learning requires a large amount of labeled data that require time-consuming and costly
1The source code is available at https://github.com/ tticoin/nkg-re.
manual efforts to construct (Yao et al., 2019; Zhang et al., 2017; Miranda et al., 2021).
To alleviate the efforts, Mintz et al. (2009) proposed distantly supervised RE (DSRE), which constructs a distantly supervised corpus in which an unlabeled corpus is automatically labeled using existing knowledge bases. Unlike any manually labeled corpus, a distantly supervised corpus is directly connected to the knowledge base entries because it is labeled based on the target properties between entities described in the knowledge bases. However, existing DSRE uses only the distantly supervised corpus and does not take advantage of other rich information in the knowledge bases, such as the features of the entities, entities that do not appear in the corpus, and a wide range of relationships between entities.
To address the limitations and utilize the rich information in the knowledge base for RE, we propose a model for biomedical DSRE from documents that constructs neighborhood knowledge graphs and integrates them into RE using the distantly supervised corpus. A neighborhood knowledge graph consists of the neighbors of the knowledge base entities appearing in an input document. When constructing neighborhood knowledge graphs, we eliminate links between target entity pairs during distantly supervised learning to avoid label leakage. By using neighbors, we can avoid processing a huge amount of information in the knowledge base in processing each entity pair.
The contributions of this paper are summarized as follows:
• We propose to construct a neighborhood knowledge graph representing the information around all of the entities in an input document.
• We build a novel distantly supervised biomedical RE model that integrates the representations of entities acquired from the neighborhood knowledge graphs with representations
363
Delete A-B edge, A-C edge
Construction depending on
the input document
A
C
B
Neighborhood Knowledge Graphs
A
C
B
Knowledge Graphs
C B A
Node Representations B E R T
GCN
Input Document drug-A reduces disease-B and disease-C ………………. reduce
affect binding promote
・ ・
・ ・
用語 関係名 用語 drug-A reduce disease-B drug-A increase disease-C
・・・ ・・・ ・・・
データベース
entity relation entity drug-A reduce disease-B drug-A affect disease-C drug-A promote disease-D ・・
データベース
・・ ・・
Database
Figure 1: Overview of the proposed model
of the input document.
• Our model achieved the performance improvement by adding the representations of the entities from the Comparative Toxicogenomics Database (CTD) (Davis et al., 2021) over the ChemDisGene dataset (Zhang et al., 2022), a document-level DSRE dataset.

heading: 2 Related work

heading: 2.1 Distantly Supervised Relation Extraction
text: DSRE was proposed by Mintz et al. (2009) to train RE models without requiring manually labeled data. Distantly supervised data is built using knowledge bases and a large-scale unlabeled corpus. A relation is labeled between two entities when the two entities registered in the knowledge base co-occur in the unlabeled corpus.
Zhang et al. (2022) proposed ChemDisGene, a document-level distantly supervised relationship extraction dataset, and evaluated a relationship extraction model. ChemDisGene assigns interactions between drugs, diseases, and genes registered in CTD, an integrated database of drug, disease, gene, mutation, and metabolism interactions, to the titles and abstracts of 80,925 documents in the MEDLINE biomedical literature database (Coordinators, 2018). The RE model consists of a pre-trained model BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019), a max pooling layer, and a fully connected layer. In addition, Wang et al. (2022) shows high performance on the ChemDisGene dataset using a positive-unlabeled (PU) learning under a prior shift of training data. They also proposed to use a squared ranking loss using a “NA” (none) class score as an adaptive threshold.

heading: 2.2 Knowledge Graph Representation Learning
text: Graph Convolutional Networks (GCNs) (Kipf and Welling, 2017) are a type of Graph Neural Networks (GNNs) that represent the structure of a graph using neural networks and update the representation of each node by convolving the representation of the target node with those of its neighbor nodes.
In addition, several methods have been proposed to represent knowledge graphs using subgraphs extracted from the neighborhood of the target nodes. For example, Learning from Subgraphs, Embeddings and Attributes for Link Prediction (SEAL) proposed in (Zhang and Chen, 2018) shows high performance in the link prediction task using GNNs. They proved that heuristically extracting subgraphs can be approximated by extracting a set of nodes within a certain number of hops. However, there is no method that uses subgraphs in the neighborhood of targets in DSRE.

heading: 3 Proposed model
text: The overview of the proposed model is displayed in Figure 1. In our study, we propose to integrate the information in the neighborhood of the entities from the knowledge graphs that are present in the input document.

heading: 3.1 Construction of Neighborhood Knowledge Graphs
text: To avoid accessing the entire knowledge base in classifying each pair in a document and to utilize the information related to the target document, we construct the neighborhood knowledge graph based on triples that are close to all the entities in an input document in the knowledge base. To obtain the close triples, we extract a set of triples that are connected to the entities within a certain number of hops. We remove the triples between the pairs
Max pooling + Concatenation
Max pooling +
Concatenation
FC
reduce affect binding promote
・ ・ ・
・ ・
relation types
that provide the labels of the distantly supervised training data to avoid label leakage. To accelerate the training, we precalculate the neighborhood knowledge graph for each document once before training the model, while in prediction, we need to generate the neighborhood knowledge graph.

heading: 3.2 Integrated use of language information and neighborhood knowledge graphs
text: We perform RE by integrating entity information of the neighborhood knowledge graphs into the distantly supervised training data. The proposed RE model is illustrated in Figure 2.
For the base RE model, we employ the BERTbased RE model in Zhang et al. (2022). Specifically, we first extract the representations corresponding to the mentions of each entity of the target pair from the final layer of BERT. We next obtain the representation of each entity through max pooling. The representations of the entities are then concatenated and classified into relational labels through a fully connected layer (FC).
When integrating neighborhood knowledge graphs, the constructed neighborhood knowledge graph is processed by GCN, and the node representations corresponding to the entities are fed to BERT along with the input document. Specifically, the node representations corresponding to the entities in each input document are added to the document while matching the position identifiers of the entities in the document and the node representations of the graph (Zhong and Chen, 2021). We obtain the representations of each graph node of the target pair by pooling the corresponding representations from the final layer of BERT corresponding to the node, and we concatenate the pooled repre-
sentations with the pooled representations of the entity mentions for classification.
GCN and BERT are trained simultaneously using Adam (Kingma and Ba, 2015) as the optimization method and the cross entropy loss.

heading: 4 Experimental Settings
text: We trained models using ChemDisGene (Zhang et al., 2022), distantly supervised data (CTDderived) as the training and development data, and manually-labeled data (Curated) as the test data. The CTD-derived data contain 76,942 and 1,521 medical references as training and development data, respectively. The Curated data contain 523 medical references as test data. We used the microaveraged F1 measure as the evaluation metric. The task is to classify entity pairs of chemical, disease, and gene into 14 relation types: two drug-disease relation types, ten drug-gene relation types, two gene-disease relation types, or no relation.
The entities of the neighborhood knowledge graphs are the drugs, diseases, and genes of CTD. The neighborhood knowledge graphs contain entities within two hops from any target entities corresponding to entities in each input document. All nodes with more than 100 edges are randomly restricted to 100 edges to reduce computation time. The statistics of the ChemDisGene and CTD are summarized in Appendix B.
The baseline model is the same as Zhang et al. (2022), excluding the neighborhood knowledge graphs from the proposed model. Following Zhang et al. (2022), we used PubMedBERT (Gu et al., 2021) for BERT. The input and output vectors are set to 768 dimensions, the maximum length of the text is 512, and the number of GCN layers is two.

heading: 5 Results
text: The performance of extracting relationships between drugs, diseases, and genes from the documents on the Curated test data is shown in Table 3. The detailed performance for each relation type is summarized in Appendix A. The proposed model (+Neighborhood KG) improved the micro-averaged F-score by 1.5 percentage points compared to the baseline (PubMedBERT). This result suggests that the information of neighborhood knowledge graphs can improve prediction performance and that the relationship extraction can take the knowledge graph information into account.
Compared to the scores of Zhang et al. (2022), our baseline PubMedBERT result showed slightly higher performance than PubMedBERT, while our PubMedBERT+BRAN model showed comparable performance, as shown in Table 3. This shows we could correctly reimplement these models.
We also incorporated our neighborhood KG into the PubMedBERT+BRAN model, but it did not show performance improvement. Furthermore, compared to the scores of Wang et al. (2022), our proposed model showed a performance lower than PubMedBERT+SSR-PU+ATLOP. However, their model does not use the neighborhood knowledge graph, and our proposed method can be incorporated into their model. We will leave the adaptation of our approach to these models for future work.
In order to examine the influence of the neigh-
borhood knowledge graph information on RE, we examined the cases that are not registered in CTD but annotated in the Curated test data. The baseline extracted 24.7 relations, whereas the proposed model extracted 40.2 relations on an average of 5 models. This result suggests that the neighborhood graph information is helpful in extracting new relations when the entities are in the knowledge base. The examples that are correctly extracted by the proposed model are shown in Table 2.

heading: 6 Conclusion
text: we proposed to integrate the information in the neighborhood of the entities from the knowledge graphs that are present in the input document. We trained and evaluated the proposed model on the ChemDisGene dataset and found that the introduction of the neighborhood knowledge graphs improved the micro-averaged F-score by 1.5 percentage points. We also confirmed that the proposed model is able to extract relationships that are not registered in the knowledge base, which could not be extracted by the model using only language information.
For future work, we will apply our model to state-of-the-art models. We will also investigate the way to utilize the extracted results in enhancing knowledge graphs to link the text and knowledge graph information more deeply and leverage the knowledge graph information more effectively.
Limitations
We have applied our neighborhood knowledge graphs to the PubMedBERT and PubMedBERT+BRAN models and show the effectiveness of the graphs on the PubMedBERT model. We have not deeply investigated how our approach cooperates with other enhancements, and the performance is lower than the state-of-the-art model (Wang et al., 2022).

heading: Acknowledgements
text: This work was supported by JSPS Grant-in-Aid for Scientific Research JP20K11962.

heading: A Detailed evaluation results
text: Table 3 shows the extraction performance of the baseline and our model for each relation type.

heading: B Dataset statistics
text: Tables 4 and 5 show the statistics of the data set.

heading: Future Work
text: For future work, we will apply our model to state-of-the-art models. We will also investigate the way to utilize the extracted results in enhancing knowledge graphs to link the text and knowledge graph information more deeply and leverage the knowledge graph information more effectively. Limitations
We have applied our neighborhood knowledge graphs to the PubMedBERT and PubMedBERT+BRAN models and show the effectiveness of the graphs on the PubMedBERT model. We have not deeply investigated how our approach cooperates with other enhancements, and the performance is lower than the state-of-the-art model (Wang et al. , 2022)