Biomedical Document Classification with Literature Graph Representations of Bibliographies and Entities
Year: 2023

Abstract:
This paper proposes a new document classification method that incorporates the representations of a literature graph created from bibliographic and entity information. Recently, document classification performance has been significantly improved with large pre-trained language models; however, there still remain documents that are difficult to classify. External information, such as bibliographic information, citation links, descriptions of entities, and medical taxonomies, has been considered one of the keys to dealing with such documents in document classification. Although several document classification methods using external information have been proposed, they only consider limited relationships, e.g., word cooccurrence and citation relationships. However, there are multiple types of external information. To overcome the limitation of the conventional use of external information, we propose a document classification model that simultaneously considers bibliographic and entity information to deeply model the relationships among documents using the representations of the literature graph. The experimental results show that our proposed method outperforms existing methods on two document classification datasets in the biomedical domain with the help of the literature graph. Our source code is publicly available at https: //github.com/tticoin/BDCL-LitGraph.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 385–395 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Document classification has improved significantly with large language models, such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019). However, these methods use only text information and ignore a lot of information behind the text, such as bibliographic information (e.g., authors and publishing journals), citation information, and information about entities that appear in the target text, which can be considered as one of the keys to better classification.
With the help of neural models, several document classification methods have been proposed that use both text information and external information. Yao et al. (2019) classified documents using representations of a text graph. In the text graph, nodes correspond to papers and words, and edges are connected between word nodes and their paper nodes and between word nodes with high co-occurrence frequency. The text graph allows paper nodes to be connected through the common word nodes and makes document classification take into account the relationships among papers. BertGCN (Lin et al., 2021) further adds text information to the node representations of the text graph using BERT. Yasunaga et al. (2022) proposed a pretrained language model LinkBERT that considers the relationship between documents. The inputs of the model are texts from documents in a citation relationship in pre-training. LinkBERT achieved higher performance than existing BERT models. Although these studies indicate the effectiveness of external information other than text information, they only consider limited information, such as the co-occurrence of words and citation relationships, and do not simultaneously consider multiple types of external information.
This paper proposes a document classification model that incorporates multiple types of external information into the target text information. Specifically, we first build the literature graph using bibliographic information, including authors and publishing journals, and entity information, including descriptions of entities and their taxonomic information. Then, we create representation vectors of the nodes that consider various relationships among different types of nodes in the literature graph so that the vectors can contain multiple types of external information. Finally, we build a document classification model that receives both the representation vectors and the target text information as input.
385
The contributions of this paper are as follows:
• Using representation vectors from the literature graph containing bibliographic and entity information, a novel document classification model is proposed that incorporates the information into the target text information.
• The proposed model with literature graph representation performs better than existing models on two document classification datasets in the biomedical domain: Ohsumed (Joachims, 1998) and Hallmarks of Cancer (HoC) (Baker et al., 2015).

heading: 2 Related work

heading: 2.1 Document classification
text: There are two types of document classification methods: methods that use only the target text information and methods that consider external information in addition to the target text information.
Yao et al. (2019) proposed a document classification model TextGCN using a text graph in which papers and words are nodes and edges weighted by TF-IDF values were connected between each paper node and its word nodes, whose words appear in the paper. In this text graph, edges were also connected between word nodes when considered highly relevant according to the PMI values. A Graph Convolutional Network (GCN) (Kipf and Welling, 2017), which aggregates information from the surrounding nodes to a node through the edges connected to the node, was used to update the node representation of the text graph with taking into account the graph structure, and the representation was used for document classification. Furthermore, BertGCN (Lin et al., 2021) achieved higher performance than TextGCN by initializing the representation vectors of the paper nodes in the text graph with BERT to incorporate text information about the papers into the graph.
Yasunaga et al. (2022) proposed LinkBERT, a pre-trained model considering the relationship between documents by using two linked documents as input to BERT simultaneously. LinkBERT was pre-trained with two tasks: masked language modeling, which was proposed in the original BERT model, and document relation prediction, which took two documents as input and classified whether they were in a citation relationship or not, aiming at modeling the information on the dependencies
between documents and the information across documents. As a result, LinkBERT outperformed existing methods on the GLUE (Wang et al., 2018) and BLURB (Gu et al., 2021) tasks, which are the benchmarks in the general and biomedical domains, and achieved higher performance than existing BERT models in document classification.

heading: 2.2 Graph representation Learning
text: Graph representation learning has been actively studied to obtain representation vectors of nodes and links from a graph by taking into account the graph structure (Hamilton, 2020). In graph representation learning, it is common to represent the graph structure as a set of triples (h, r, t) using the head h, the tail t and the relation r of a directed edge, and methods such as TransE (Bordes et al., 2013) and DistMult (Yang et al., 2015) have been proposed to learn to represent the nodes and links in these triples. For instance, TransE uses the distance between h + r and t as the score function. TransE is a simple and effective method, but it has several problems. For example, TransE cannot represent the relations that have multiple tail nodes for a head node. There are several models such as TransH (Wang et al., 2014) and TransR (Lin et al., 2015) that address these limitations. These models project the node representations into the relation-specific space using projections for each relation.
RotatE (Sun et al., 2019) is capable of modeling various relation patterns, including symmetry/antisymmetry, inversion, and composition. In RotatE, each relation is defined as a rotation from the head node to the tail node in the complex space.

heading: 3 Proposed method
text: This section proposes a novel document classification model that incorporates a representation of a literature graph with bibliographic and entity information. Figure 1 shows an overview of the proposed method. We first explain the definition and representation learning of the literature graph in Sections 3.1 and 3.2, respectively. We then introduce a document classification model that uses both vector representations of the literature graph containing bibliographic and entity information, and the target text information is described in Section 3.3.

heading: 3.1 Definition of the literature graph
text: We define a literature graph based on bibliographic information and entity information. The literature graph has the papers, authors, publication years, publication venues, and entities as nodes. Each paper node is connected to other types of nodes based on bibliographic information about the paper. Each paper node is also connected to other paper nodes when they have a citation relationship. Furthermore, the entity nodes are connected if two entity nodes are in a hypernym-hyponym relationship or a supplementary concept relationship (Figure 1(1)).
The title and abstract are added to each paper node as text information, and the entity description providing the scope and content of the entity, which is given as “ScopeNote” in the entity database, is added to each entity node if it exists.

heading: 3.2 Representation learning on literature graph
text: First, the paper and entity nodes are initialized with BERT using their text information (Figure 1(2)). The representation of the [CLS] token in the BERT output is used as the initial representation of the corresponding node, since it is considered the repre-
sentation of the whole sentence. If the text information is unavailable, the node is randomly initialized so that it follows a normal distribution of the mean and standard deviation of the node representations initialized by BERT. Other types of nodes are also randomly initialized.
Then, representation learning is performed on the literature graph using TransE (Bordes et al., 2013)1 to obtain the vector representation of the literature graph (Figure 1(3)). The vector representation is expected to take into account various relationships between documents based on bibliographic and entity information, as well as the target text information.

heading: 3.3 Document classification with literature graph representation
text: To use the information from the literature graph in document classification, in addition to the target text information, the vector representations corre-
1TransE is employed because it is a simple model that deals with representations in the Euclidean space and the literature graph treated in this study is large and computationally expensive methods such as GCN (Kipf and Welling, 2017) cannot be easily applied. The application of other methods is left for future work.
sponding to the target paper in the literature graph obtained in Section 3.2 are used (Figure 1(4)). For this purpose, the vector representations of the target paper node and entity nodes corresponding to the entities appearing in the paper denoted Plg and Elg are used.
The proposed model is shown in Figure 2. The entities in the target text are first extracted by caseinsensitive string matching with the entities registered in the database. The paper node representation and the entity node representations, which correspond to the entities in the target text, are then appended to the target text information. The input of BERT when M entities are extracted from a paper i is as shown in the following Equation (1).
S = {[CLS],w1, · · · , wn, [Pi], [E1], · · · , [EM ]}
(1)
where wi is a subword in the target text, Pi is the tokens representing the paper i’s node of the literature graph, and Ei is a token representing the entity i’s node of the literature graph. At this time, the tokens representing the nodes in the literature graph are mapped to the corresponding tokens in the text using the position IDs (Zhong and Chen, 2021); the same position ID is assigned to the [CLS] token and the token representing the paper node. Similarly, the same position ID is assigned to the first subword of the entity in the target text and the token representing the entity node. To allow appending
as many paper and entity nodes as possible, the target text is truncated if it exceeds the BERT max length of 512 tokens. Also, if the beginning of an entity does not match the beginning of a subword, the representation of that entity is not used. The representations of [CLS] and wi are assigned from the pre-trained BERT embedding table, the representations of Pi and Ei are assigned representation of the literature graph as follows:
W0 = {w[CLS],ww1 , · · · ,wwn , Plg(Pi),Elg(E1), · · ·Elg(EM )}, (2)
where w[CLS] and wwi are representations of [CLS] and wi, respectively. Plg(Pi) and Elg(Ei) are representation of Pi and Ei of the literature graph. W0 is used as the input to BERT, and the representations of the final layer of BERT are obtained as in Equation 3.
Wl+1 = Self-attentionl(Wl) (3)
The representations of the final layer of BERT are represented as Equation 4. The representations among them corresponding to the [CLS] token and the paper node, as well as the paper representation of the literature graph, are concatenated to create hpaper(i) and used for classification using a fully connected (FC) layer. The output zpaper(i) from the FC layer is converted to probabilities by applying the softmax function for single-label data and the sigmoid function for multi-label data.
WL = {h[CLS],hw1 , · · · ,hwn , PLlg(Pi),E L lg(E1) , · · · , ELlg(EM )} (4)
hpaper(i) = [h[CLS];P L lg(Pi) ;Plg(Pi)] zpaper(i) = FC(hpaper(i)) (5)

heading: 4 Experimental Settings
text: The quality of the representation vectors obtained by representation learning of the literature graph is evaluated by link prediction, which predicts the target nodes that are related to the node. We also evaluated document classification using the representation of the literature graph to confirm the effectiveness of the literature graph in document classification. We used BioLinkBERT-base (Yasunaga et al., 2022) in both experiments.

heading: 4.1 Representation learning of the literature graph
text: The 2022 version of the medical literature database MEDLINE (National library of Medicine, 2020) was used to create the literature graph described in Section 3.1. The literature graph is huge because over 30 million articles are registered in MEDLINE, so author nodes that had few connections to other nodes with a degree of less than five
were deleted. The 2021 version of MeSH (National Library of Medicine, 2020) was used as the entity database. Each paper was assigned with the MeSH as entities. We used only MeSH entities representing the most significant points labeled as “Major Topic” in MEDLINE for the MeSH relation in our literature graph. The literature graph has edges representing citation relations (Cites), between documents and their authors (Author), their years of publications (Year), their publication journals (Journal), and their entities (MeSH), hypernym-hyponym relations between entities (Hypernym), and supplementary concept relations between entities and supplementary MeSH entities (Supp). The statistics of nodes and edges of the literature graph are shown in Tables 1 and 2, respectively.
MAP@30 and Hit@N were employed as the evaluation metrics for link prediction on the literature graph. Data were split into a ratio of 98:1:1 for training, development, and test data sets, keeping the same ratio of relationship types. The development and test triples were chosen so that the nodes in the triples appear in the training data set. For the evaluation of link prediction, the target nodes included in the training data sets were removed from the prediction candidates. Only nodes with a target node type determined from the head and a relation type were used for prediction; for example, in predicting the Author relation from a paper node, the Author nodes were used for the prediction candidates. Section 4.3 shows the libraries and training setups used for the experiments.
For comparison, we build an entity graph that is a subgraph of the literature graph and has only entity nodes. The entity graph comprises hypernymhyponym relationships and supplementary concept relationships among entities, so the entity node representations from the entity graph reflect the re-
lationships between entities. Entity graph representation is learned in the same setting as the literature graph, and the obtained entity node representations Eeg are used for document classification.

heading: 4.2 Document classification using the representation of the literature graph
text: For the evaluation, we used Ohsumed (Joachims, 1998) and Hallmarks of Cancer (HoC) (Baker et al., 2015). Ohsumed is a document classification dataset composed of abstracts in the biomedical domain, in which documents are assigned one or more of 23 different cardiovascular disease categories. Since Ohsumed is built using MeSH, the relationships between the papers and MeSH in Ohsumed were excluded from the literature graph, and this literature graph was used for both datasets. As in existing studies (Yao et al., 2019), documents with multiple categories were excluded. This resulted in 3,357 and 4,043 documents in the training and test data sets, respectively. The training data set was divided into 7:3 to create a development data set. Each document in the HoC was assigned multiple categories chosen from 10 different cancer features.
HoC was split according to existing studies (Gu et al., 2021), and we used 1,295, 186, and 371 documents for training, development, and test data sets, respectively. Both datasets are suitable for evaluating our proposed method as they are assigned categories that are semantically close and difficult to classify with textual information. For both data, the entities in the papers were extracted by string matching with the entities in the database, resulting in 20.49 entities per paper for Ohsumed and 26.94 entities per paper for HoC on average.
In the evaluation, three models with different seeds of random numbers were trained for the Ohsumed, and five models for the HoC, and the average of their evaluation scores are reported as the final predicted result; accuracy was used for the evaluation of Ohsumed and the F1-measure for the evaluation of HoC. As a baseline model, we modified the document classification model in Section 3.3 to use only textual information as input. We compared the settings with or without paper and entity representations from the literature graph. We also compare the entity information from the literature and the entity graphs. When both entity information is used, each representation is added to the target text, and the same position ID is assigned to each representation and the first subword of the entity in the target text.
We chose the best setting for the final test on each corpus.

heading: 4.3 Experimental environments
text: Python 3.7.11 was used for implementation, DGLKE 0.1.2 (Zheng et al., 2020) for TransE, Transformers 4.19.4 (Wolf et al., 2020) for using the pretraining model, and Pytorch 1.10.0 (Paszke et al., 2019) for model creation. The link prediction evaluation was approximated using the neighborhood search library NGT (Iwasaki and Miyazaki, 2018).
TransE was trained 50 epochs for both the literature graph and entity graph. As the representations of paper and entity nodes in the literature graph are initialized with BERT, the representation is 768- dimensional, so the representation of nodes to be randomly initialized was also initialized with 768 dimensions. Negative sampling, which randomly replaces h or t of the triple (h, r, t) on the graph, is used in TransE training. Only nodes with a target node type determined from the head node type and the relation type were used for negative sampling. A TransE training was conducted on an AMD Ryzen Threadripper 3990X 64-Core Processor as CPU and a GeForce RTX 3090 as GPU.
In the document classification model, the representation of the [CLS] token in BERT is classified with a linear layer. A dropout (Srivastava et al., 2014) was added before the linear layer to prevent overfitting on the training data. The document classification was performed on Intel(R) Xeon(R) CPU E5-2698 v4 and Intel(R) Xeon(R) W-3225 CPU as CPUs and Tesla V100-DGXS-32GB and A100. AdamW (Loshchilov and Hutter, 2019) was used as the optimizer for the document classification model.

heading: 5 Results

heading: 5.1 Representation learning
text: The results of link prediction on the literature graph are summarized in Table 3. Since the literature graph used in this study had a large number of nodes, both MAP@30 and Hit@N were generally low. The low performance of the relation between papers (Cites) and the relation between MeSHs (Hypernym, Supp) in particular may be due to the fact that TransE cannot represent them because there can be multiple tail nodes for a head node. Although the performance is low for link prediction, since the node representation is prepared for document classification, not link prediction, it would not be a critical problem if the representation cannot represent the multiple tail nodes.
The comparison of the link prediction results
of the literature graph and entity graph is shown in Table 4. For the hypernym-hyponym relations (Hypernym), the performance was higher on the literature graph than on the entity graph. On the other hand, for the supplementary concept relations (Supp), the relations could be predicted only on the entity graph. MeSH consists of two types of concept records: descriptors and supplementary concept records. The descriptors play a central role in MeSH and have a tree structure. In addition, each paper from Medline is assigned only the descriptors as the entity representing the paper. On the other hand, supplementary records do not have a tree structure and are linked to descriptors. Therefore, in the literature graph, descriptors have more relationships than supplementary records. Thus the Hypernym relations, which are relationships between descriptors, may be taken into account more in the literature graph.

heading: 5.2 Document classification
text: The comparison of our model with the existing models on the test set is summarized in Table 5. The scores are taken from the original papers, with the exception of the result of BioLinkBERT on Ohsumed and our results. For our model, we show the results with the best setting on the development set, i.e., adding Plg for Ohsumed and Elg for HoC. The tuning results on the development data sets
are shown in Table 6. The classification scores are higher than the previously reported scores, especially for Ohsumed.
As in Table 6, when we added each type of node representation from the literature graph separately, the performance improved over the baseline for both datasets. As for the entity representations, the entity representations from the literature graph are more effective than those from the entity graph. Especially on the Ohsumed dataset, adding the entity representations from the entity graph shows a negative effect on the performance. When we use multiple types of representations, the performance is not better than that of a single type of representations. The results show that the simultaneous use of multiple types of representations is not simple.

heading: 6 Analysis
text: To check the effects of the representation vectors of the literature graph and the entity graph alone, we conducted two experiments: document classification with masking the entities in the target text, which eliminates the impact of entity information in the text, and document classification with no text information, which eliminates the impact of paper and entity information in the text. The overview of these experiments is shown in Figure 3. The results of each experiment are shown in Table 7. Note that in these experiments, only the representation of the final layer of BERT corresponding to the [CLS] is used for classification. Masking entities in the target text. As shown in Table 7, when the entities in the target text are masked, the performance is degraded for all settings. This may be because the entity information in the language model is not available. We can
.
see the use of Elg is effective by comparing the settings with or without Elg. From this result, we can say that Elg compensates for the missing entity information caused by masking entities. The performance with Eeg is lower than one with Elg, which is consistent with the results in Table 6, suggesting that the Eeg representation is less suitable for document classification compared to Elg. This may partly be because that Elg takes into account the relationship of entities with documents instead of independently representing entities as for Eeg. No text information. To evaluate the performance only with the representations obtained from the graphs, we excluded the target text from the BERT input, i.e., only paper and entity representations were used as input to BERT. We found that even the paper and entity representations from graphs alone could classify documents to some extent despite the lack of text information. The performance of using both the paper and entity representations was higher than that of using each representation. These
results show that the paper and entity information from literature and entity graphs are all helpful in classifying documents, although combining them with text information is not always helpful.

heading: 7 Conclusions
text: In this study, for the purpose of document classification that can use multiple types of information at the same time, we proposed a document classification model that creates a representation vector from a literature graph that contains a lot of information, such as bibliographic and entity information, considering various relationships between documents, and incorporates the representation vector and textual information about the documents. Experimental results on two datasets, Ohsumed and HoC, show that both models improve performance with the information from the literature graph, and the models show state-of-the-art performance on both datasets. We also found that the performance degraded when we simultaneously used multiple information of paper and entity, showing that the incorporation of different types of information is not simple.
In the future, we will investigate representation learning of large-scale literature graphs using methods such as GCN to obtain better representations. We will also explore methods for simultaneously learning representations of literature graphs and document classification.
Limitations
Our proposed model has three limitations. Firstly, because of the large size of the literature graph in this study, representation learning is performed on the literature graph using TransE, but in fact, there are relations in the literature graph that cannot be represented by TransE as shown in Section 5.1. To overcome this issue, more expressive methods such as RotatE (Sun et al., 2019) and GCN (Kipf and Welling, 2017) could be investigated. These methods are expected to be able to represent complex relationships associated with multiple types of external information. Secondly, our model adds a representation of the literature graph to the target text, so longer sentences require truncation of textual information. Thirdly, we have not analyzed the results in detail and how the proposed method positively and negatively impacted document classification. We leave these limitations for future work.

heading: Acknowledgements
text: This work was supported by JSPS Grant-in-Aid for Scientific Research JP20K11962.

heading: A Tuning results on the development data set
text: The tuning results on document classification on the development data set are shown in Table 8. The representations used for the inputs were compared as in Table 6, and tuning was performed to determine which combination of the representations of the final layer of BERT corresponding to the [CLS] token and the paper node, as well as the paper representation of the literature graph is used for classification using the FC layer. We also tuned whether the representations used for classification should be concatenated (concat) or pooled with max pooling.

heading: Future Work
text: In the future, we will investigate representation learning of large-scale literature graphs using methods such as GCN to obtain better representations. We will also explore methods for simultaneously learning representations of literature graphs and document classification. Limitations
Our proposed model has three limitations. Firstly, because of the large size of the literature graph in this study, representation learning is performed on the literature graph using TransE, but in fact, there are relations in the literature graph that cannot be represented by TransE as shown in Section 5. 1. To overcome this issue, more expressive methods such as RotatE (Sun et al. , 2019) and GCN (Kipf and Welling, 2017) could be investigated. These methods are expected to be able to represent complex relationships associated with multiple types of external information. Secondly, our model adds a representation of the literature graph to the target text, so longer sentences require truncation of textual information. Thirdly, we have not analyzed the results in detail and how the proposed method positively and negatively impacted document classification. We leave these limitations for future work