JMIR MEDICAL INFORMATICS

Wang et al

Original Paper

Document-Level Biomedical Relation Extraction Using Graph
Convolutional Network and Multihead Attention: Algorithm
Development and Validation
Jian Wang1, PhD; Xiaoyu Chen1, BSc; Yu Zhang1, MSc; Yijia Zhang1, PhD; Jiabin Wen2, PhD; Hongfei Lin1, PhD;
Zhihao Yang1, PhD; Xin Wang1, BSc
1

School of Computer Science and Technology, Dalian University of Technology, Dalian, China

2

Department of VIP, The Second Hospital of Dalian Medical University, Dalian, China

Corresponding Author:
Yijia Zhang, PhD
School of Computer Science and Technology
Dalian University of Technology
No. 2 Linggong Road, Ganjingzi District
Dalian, 116023
China
Phone: 86 0411 84708498
Email: zhyj@dlut.edu.cn

Abstract
Background: Automatically extracting relations between chemicals and diseases plays an important role in biomedical text
mining. Chemical-disease relation (CDR) extraction aims at extracting complex semantic relationships between entities indocuments, which contain intrasentence and intersentence relations. Most previous methods did not consider dependency syntactic
information across the sentences, which are very valuable for the relations extraction task, in particular, for extracting theintersentence relations accurately.
Objective: In this paper, we propose a novel end-to-end neural network based on the graph convolutional network (GCN) andmultihead attention, which makes use of the dependency syntactic information across the sentences to improve CDR extraction
task.
Methods: To improve the performance of intersentence relation extraction, we constructed a document-level dependency graphto capture the dependency syntactic information across sentences. GCN is applied to capture the feature representation of the
document-level dependency graph. The multihead attention mechanism is employed to learn the relatively important context
features from different semantic subspaces. To enhance the input representation, the deep context representation is used in ourmodel instead of traditional word embedding.
Results: We evaluate our method on CDR corpus. The experimental results show that our method achieves an F-measure of
63.5%, which is superior to other state-of-the-art methods. In the intrasentence level, our method achieves a precision, recall, andF-measure of 59.1%, 81.5%, and 68.5%, respectively. In the intersentence level, our method achieves a precision, recall, andF-measure of 47.8%, 52.2%, and 49.9%, respectively.
Conclusions: The GCN model can effectively exploit the across sentence dependency information to improve the performanceof intersentence CDR extraction. Both the deep context representation and multihead attention are helpful in the CDR extraction
task.
(JMIR Med Inform 2020;8(7):e17638) doi: 10.2196/17638
KEYWORDS
biomedical relation extraction; dependency graph; multihead attention; graph convolutional networkhttps://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Introduction
Valuable biomedical information and knowledge are still hiddenin the exponentially increasing biomedical literature, such as
the chemical-disease relation (CDR). Extracting the relation
between chemicals and diseases is an important task inbiomedical text mining, which plays an important role in various
biomedical research studies, such as clinical treatment, drug
development, and biomedical knowledge discovery [1-3].
However, extracting CDR from the biomedical literature
manually is time-consuming and difficult to keep up-to-date.
Thus, the BioCreative V community [4] proposed a task ofextracting CDR in the biomedical literature automatically topromote the research on the CDR extraction.
To date, many methods have been proposed for automatic
relation extraction between chemicals and diseases, which canbe divided into 3 categories: rule-based methods [5],feature-based methods [6-9], and deep neural network-based
methods [10-13]. Rule-based methods aim to formulate theheuristic rules for CDR extraction. Lowe et al [5] developed apattern-based system with some heuristic rules to extract
chemical-induced disease (CID) relations within the samesentence. The heuristic rules are used to extract the most likely
CID relations when no patterns match a document. Generally,rule-based methods are simple and effective. However, thesemethods are difficult for application in a new task or dataset.
Feature-based methods aim at designing rich features, including
semantic and syntactic information. Xu et al [6] utilized text
features, including context information and entity information,
incorporated with domain knowledge to extract CID relations.
Since the syntactic information carried in the dependency graphof the sentence is crucial to CDR extraction, some studies also
developed syntactic features. Gu et al [7] utilized various
linguistic features to extract CID relations with the maximum
entropy model. They leveraged lexical features for both
intrasentence and intersentence level relation extraction anddeveloped the dependency features only for intrasentence level
relation extraction. Zhou et al [8] utilized the shortest
dependency path between chemical and disease entities toextract structured syntactic features. Feature-based methods
achieve better performance than rule-based methods. However,traditional feature-based methods only use the dependency trees
to extract local syntactic dependencies for the intrasentencelevel relation extraction, without considering the syntactic
dependencies across sentences for the document-level relation
extraction. Besides, designing rich features is a time-consuming
and laborious task.
In recent years, the deep neural network has been widely usedin various natural language processing (NLP) tasks. Somestudies have developed deep neural network-based methods forbiomedical relation extraction. Long short-term memory(LSTM) models and convolutional neural network (CNN)
models are the 2 major neural networks. Zhou et al [10] applied
LSTM and CNN models based on traditional word embedding
to capture context features for CDR extraction and achieve agood performance. Gu et al [11] proposed a CNN-based model
to capture context and dependency features for intrasentencelevel relation extraction. Nguyen and Verspoor [13] investigatedhttps://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

Wang et alcharacter-based word embedding into the CNN-based relation
extraction model. Traditional word embedding such as word2vec
cannot vary according to linguistic contexts effectively. Peters
et al [14] proposed deep contextualized word representations
called ELMo based on a deep bidirectional language model.
ELMo can generate a more comprehensive representation foreach word based on the sentence context. Therefore, integrating
ELMo with a deep neural network may improve the performanceof CDR extraction.
In both CNN-based and LSTM-based models, it is hard to
distinguish the relevant and irrelevant context features for the
relation extraction. A recent study [15] suggested that attention
mechanism can capture the most important semantic information
for the relation extraction. Vaswani et al [16] introduced amultihead attention mechanism that applied the self-attention
mechanism multiple times to capture the relatively importantfeatures from different representation subspaces. Thus,
multihead attention mechanism can be used to improve theperformance of the CDR extraction.
Dependency trees are often used to extract local dependencies
for intrasentence level CDR extraction. However, existingstudies ignored the nonlocal dependency across sentences, whichis crucial for intersentence level CDR extraction. Quirk et al[17] introduced a document graph that can derive features within
and across sentences. Thus, we also constructed adocument-level dependency graph that can extract dependencies
for intrasentence and intersentence level CDR extraction
simultaneously. Recently, the graph convolution network (GCN)[18] has been effectively used for encoding document graphinformation. Thus, GCN can operate directly on the
document-level dependency graph to capture long-range
syntactic information, which is useful for CDR extraction.
In this study, we evaluated the effectiveness of the deep
contextualized word representations, multihead attention
mechanism, and GCN in the CDR extraction task. To improve
the performance of the intersentence relation extraction, weconstructed the document-level dependency graph to capture
the dependency syntactic information across sentences. Basedon the document-level dependency graph, we proposed a novel
end-to-end model to extract CID relations from the biomedical
literature. First, we used ELMo, POS embedding, and position
embedding to construct the input representation and employed
the multihead attention with bidirectional LSTM (BiLSTM) to
capture the relatively important context features. Second, we
employed the GCN to capture the long-range dependency
features based on the document-level dependency graph. Third,
we combined the context features and long-range dependency
features as the final feature representation and applied a Softmax
function to implement relation classification. Finally, weevaluated our model on the CDR corpus.

Methods
CDR Extraction
The CDR extraction task is a challenging task, which was
proposed by the BioCreative V community. The CDR extraction
task aims to extract CDR from the biomedical literature
JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
automatically and accurately. It is composed of 2 subtasks: (1)disease named entity recognition and normalization and (2) CID
relation extraction.
In this study, we focused on the CID relation extraction task.
The CDR extraction task is a document-level biomedical relation
extraction problem, which is different from traditional
biomedical relation extraction task. Traditional biomedical
relation extraction only considers relation within a single
sentence such as protein-protein interaction [19] and drug-druginteraction [20]. However, the CID relation is not only expressed
within a single sentence, but it is also expressed across several
sentences. Figure 1 shows an illustration of CDR extraction. It

Wang et al
is extracted from the CDR corpus whose PMID is 6203632.
Among these sentences, the texts in bold mention the chemicaland disease entities. In Figure 1, we mark the corresponding
entity type and the medical subject headings concept identifiers[21] after the entity mention in the sentence. The chemical
D007545 has 2 intrasentence level co-occurrences with disease
D006332 in the sentence 1 and the sentence 2, while it has anintersentence level co-occurrence with disease D006965.
However, not all occurrences of the chemicals and diseases are
considered as a CID relation. For example, the chemical
D007545 does not have a CID relation with the disease D006984
in the sentence 4 because the concept of the disease D006984
is too general to reflect a CID relation.

Figure 1. Illustrative examples of CID relation. CID: chemical-induced disease.

Relation Instance Construction
First, we should construct relation instances for both training
and testing stages. All the instances generated from the diseaseand chemical mentions in the document are pooled into 2 groups
at the intrasentence and intersentence levels, respectively. The
former means that a chemical-disease mention pair is in thesame sentence. The latter means that a mention pair is in adifferent sentence. If the relation between the chemical anddisease entity of the mentioned pair is annotated as a CID
relation in the document, then this mentioned pair is constructed
as a positive instance; otherwise, this mentioned pair isconstructed as a negative instance. We applied several effective
heuristic rules for both intrasentence and intersentence levelinstances. The details are as follows.

Relation Instance Construction for Intrasentence Level
1.

2.

3.

All chemical-disease entity mention pairs that appear in thesame sentence are constructed as intrasentence levelinstances.
If multiple mentions refer to the same entity in a sentence,
the mentions in the nearest distance should be constructed
as an instance.
For instance, chemical D007545 and disease D006332 in
sentence 1 form an intrasentence level positive instance,
while chemical D007545 and disease D006984 in sentence
4 form an intrasentence level negative instance.

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

Relation Instance Construction for Intersentence Level
1.

2.

Only the chemical-disease entity pairs that are not involvedin any intrasentence level are considered as intersentencelevel instances.
If multiple mentions refer to the same entity, the chemicaland disease mention in the nearest distance are chosen.

According to our heuristic rules, chemical D007545 in sentence
4 and disease D006965 in sentence 5 are regarded as anintersentence level instance because there are no mentions ofthem in the same sentence. Chemical D007545 in sentence 1
and disease D006965 in sentence 5 will be omitted because
their distance is not the shortest. Further, chemical D007545 in
sentence 4 and disease D006984 in sentence 5 are not regarded
as an intersentence level instance because chemical D007545
already has intrasentence level co-occurrence with disease
D006984 in sentence 4.

Document-Level Dependency Graph
To generate features for entity pairs within and across sentences,
we introduce a document-level dependency graph with nodes
representing words and edges that show intrasentence andintersentence dependency relations. Figure 2 shows an example
of document-level dependency graph for 2 sentences. In thisstudy, we use the following 3 types of intrasentence andintersentence dependency edges.

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wang et al

Figure 2. An example of a document-level dependency graph for 2 sentences expressing a CID relation. The chemical and disease entity mention ishighlighted in bold. For simplicity, we have omitted self-node edges. CID: chemical-induced disease.

1.

2.

3.

Syntactic dependency edge: The syntactic structure is
crucial to biomedical relation extraction. Hence, we use
syntactic dependency edges derived from Stanford
dependency syntactic parser as intrasentential edges. For
instance, “conj” denotes the syntactic relation between theword “stopped” and “followed” in the same sentence.
Adjacent sentence edge: Dependencies between sentences
are useful for document-level relation extraction. Thus, weconsider the sentence as a node in a type of discourse
dependency tree. Moreover, we added an edge between thedependency roots of adjacent sentences as an intersentential
edge, which is a simple but an effective approach. For
instance, “next” denotes the syntactic relation between 2
sentences.
Self-node edge: We added self-node edges to all the nodesof the graph in order to enable GCN to not only learn
information based on neighbor nodes but also learn thenode information itself.

Model Architecture

representation layer, the BiLSTM layer, the multihead attention
layer, and the GCN layer. The inputs of our model are text
sequences. The input layer will generate a deep contextualized
word representation for each word. Recent studies [22,23] havesuggested that the part of speech (POS) and the position of eachword are useful for biomedical relation extraction. Hence, we
concatenate the deep contextualized word representation and
POS and position embedding as the whole word representation.
The BiLSTM layer will obtain contextual features from theword representation. The multihead attention layer will apply
the self-attention mechanism multiple times to capture therelative semantic features from different representation
subspaces. The GCN layer will operate over the document-level
dependency graph to capture long-range syntactic features. We
employed max pooling over the outputs of the multihead
attention layer and the GCN layer and then concatenated these2 vectors as the final representation. Finally, we employed afully connected layer and the Softmax function to identify the
CID relation. Our model will be described in detail in thefollowing section.

The schematic overview of our model is shown in Figure 3. In
short, our model mainly consists of 4 parts: the input

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 4
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wang et al

Figure 3. Overview of our model. The input representation consists of ELMo, POS embedding, and position embedding. In the multi-head self-attention
layer, we only show the detailed self-attention computation for the word “administration.” In the GCN layer, we only show the detailed graph convolution
computation for the word “administration.” BiLSTM: bidirectional long short-term memory; POS: part of speech; GCN: graph convolutional network.

Input Representation
We used ELMo instead of the traditional word representationin our model. Traditional word representation generates a fixed
representation vector for the same word. However, ELMo isthe function of the entire input sentence based on a bidirectional
language model so that it can generate different representation
vectors for the same word according to the different sentence
context.
Given that a sequence {t1, t2,…..tN} denotes the word tokens in
a sentence S. Given a token tk, the forward language model
calculates the probability of the token tk based on the previous
tokens {t1, t2,..., t(k-1)} of tk in the sentence S as follows:(1)
Similarly, the backward language model calculates theprobability of the token tk based on the back tokens {t1, t2, …,
t(k-1)} of tk in the sentence S as follows:(2)

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

Combining the forward and the backward language models asa bidirectional language model, the log-likelihood can be
maximized as follows:(3)
ELMo can represent the semantic and syntactic information of
the word. In our model, we use a linear combination of thehidden state in each layer of the bidirectional language model
to generate a deep contextualized representation for words. The
POS and the position information of a word are crucial tobiomedical relation extraction. Therefore, we also utilize POS
embedding and position embedding to enhance therepresentation ability of the input. The POS embedding
represents the POS feature of a word, and the position
embedding reflects the relative distance between the word andthe target entity. Given a word at position i, we obtain its POS
embedding wp,i and position embedding wd,i based on mapping
matrixes Mp and Md, respectively. Finally, the whole word
representations concatenate deep contextualized word
representations, POS embedding, and position embedding asfollows:
wi=[we,i; wp,i; wd,i] (4)JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 5
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSBiLSTM
The LSTM model is a variant of recurrent neural networkmodels that has been used in many NLP tasks successfully. The
LSTM model overcomes the vanishing gradient problem byintroducing a gating mechanism [24]. Therefore, it is suitableto capture the long-term dependency feature. The LSTM unit
consists of 3 components: the input gate it, the forget gate ft,
and the output gate ot. At the time step t, the LSTM unit utilizes
the input word xt, the previous hidden state h(t-1), and theprevious cell state c(t-1) to calculate the current hidden state ht
and cell state ct. The equations are as follows:
ft=σ(Wfxt+Ufh(t-1)+bf) (5)ot=σ(Woxt+Uoh(t-1)+bo) (6)gt=tanh(Wgxt+Ugh(t-1)+bg) (7)it=σ(Wixt+Uih(t-1)+bi) (8)ct=ft⊙c(t-1)+ it⊙gt (9)ht= ot⊙tanh(ct) (10)where W, U, b are the weight and bias parameters, and ⊙denotes element-wise multiplication. In this study, we use the
BiLSTM model that can capture the forward and backward
context features simultaneously. The BiLSTM model combines
a forward LSTM and a backward LSTM. Given the hidden stateof the forward LSTM
LSTM

and the hidden state of the backward, the final hidden state is concatenated as:

Wang et al
where Q, K, V∈Rn represent query, key, and value matrixes,
respectively. d is the dimension of the output of the BiLSTM
model.
The main idea of the multihead attention is applying theself-attention mechanism multiple times. If the multihead
attention contains h heads, the i-th attention head can becalculated as headi=Attention (Qi,Ki,Vi). Thus, the final
multiheadattentionistheconcatenationof{head1,head2,...,headh} as MultiHead (Q,K,V)=Concat(head1,head2,...,headh) Wo. The output of the multihead attention
layer is a matrix of Rnat.

GCN
GCN is an adaptation of CNN [26], which operates on graphs.
Given a graph with n nodes, the graph structure can be
represented as an adjacency matrix A. In this study, we
converted the document-level dependency graph into its
corresponding adjacency matrix A, where Aij=1 if there is adependency edge going from token i to token j; otherwise Aij=0.
The dependency graph can be calculated as an undirected graph[27], which means Aij=Aji. Further, we add a self-node edge toall the nodes in the graph, which means Aii=1. Since the degreeof a node in the dependency graph varies a lot, this may bias
the output representation toward favoring high-degree nodes,
regardless of the information carried in the node. To solve thisissue, we normalize the activations in the graph convolution
before feeding it through the nonlinearity. Finally, the graphconvolution operation for node i at the l-th layer whereanddenote the input representation and the output representationof node can be defined as follows:

Multihead Attention
The BiLSTM model learns the context features from the input
sequences automatically and effectively. However, these features
make different contributions to the biomedical relation
extraction. In our model, we capture the relatively importantfeatures by introducing multihead attention mechanism. The
essence of multihead attention is applying self-attention
mechanism multiple times so that it may let the model learn therelatively important features from different representation
subspaces. The self-attention mechanism generates the output
based on a query and a set of key-value pairs. The output is theweighted sum of the values, where the weight assigned to eachvalue is computed by applying attention function to the querywith the corresponding key. In our study, we deal with the outputof the BiLSTM model by multihead self-attention. Further, we
use the dot-product attention function instead of the standard
additive attention function [25] as follows:(12),
where W(l) is the weight matrix, b(l) is the bias vector,

is the degree of node i in the dependency graph, and ρ is anactivation function (eg, a rectified linear unit).
The GCN model takes the output of the BiLSTM model as the
input word representation:

Then, we stack the graph convolution operation over layers andobtain(11),

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wang et alas the output word representations of the GCN model. Note that
the GCN model presented above uses the same parameters forall edges in the dependency graph.

classification. Then, the final representation is fed into a 2-layer
perceptron as follows:

Relation Classification
To make use of the output word representation of the GCN
model for relation extraction, we generate the sentence
representation as follows:
hsent=f (h(L))=f (GCN(h(0)) (13)where h(L) denotes the output representations at the last layer Lof the GCN model, and f:Rn:R→Rd is a max-pooling functionthat maps n output vectors to the sentence vector.
Inspired by recent studies [28,29], entity information is central
to relation classification. Therefore, we also obtain the chemical
entity representation hc as shown in. Similarly, we canobtain the disease entity representation hd. The feature
representation of the whole GCN model is hGCN=[hsent; hc; hd].
We also obtain the feature representation hatt from the outputof the multihead attention layer by applying max pooling to themultihead attention matrix. We concatenate hGCN and hatt toform the final representation hfinal=[hGCN; hatt] for relation(14) and (15).
Finally, the hidden representation h2 is fed to a Softmax function
to calculate the confidence of the CID relation:o=softmax (Woh2+bo) (16)where o is the output, Wo is the weight matrix, and bo is the bias
vector.

Results
DatasetWe evaluated our model on the CDR corpus, which was releasedby the BioCreative V task. The CDR dataset is the benchmark
dataset for the CID relation extraction task, which consists of1500 PubMed abstracts—500 each for training, development,
and test set. Table 1 shows the details of the dataset.

Table 1. Statistics of the chemical-disease relation dataset.
Task dataset

Abstracts (n=1500)

Chemical-induced disease relations (n=3116)

Training

500

1038

Development

500

1012

Test

500

1066

In this study, the gold entity annotations provided by
BioCreative V were used to evaluate our model. All thecomparison methods reported in this paper were evaluated withgold entity annotations. Therefore, it is fair and comparable.
Further, we measured the CID relation extraction performancewith precision, recall, and F-measure.

trained by Adam [30] with a learning rate of 0.001 and aminibatch size of 32. In addition, our model was implemented
based on an open-source deep learning library PyTorch [31].
We used StanfordNLP [32] to obtain the POS of the word andthe dependency tree. Further, we used the pretrained ELMo
representations for the deep contextualized word representations.

Experimental Settings

Experimental Results

The dimensions of POS embedding and position embedding
are both 100. The dimension of ELMo is 1024. The dimensionsof the LSTM hidden layer and the GCN layer are 500 with the
dropout proportion of 0.5. The dimensions of 2-layer perceptron
are also 500 with the dropout proportion of 0.5. Our model was

Effect of Input Representation

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

We evaluated the effectiveness of the input representation of
our model. We used the same model that we proposed andchanged the input representations. The comparison performanceof the different input representations is presented in Table 2.

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wang et al

Table 2. The effect of the input representation on performance.

a

Input representation

Precision (%)

Recall (%)

F-measure (%)

Worda

47.3

71.7

57.0

Word+positionb

49.1

71.4

58.2

Word+position+POSc

51.6

71.8

60.1

ELMod

57.0

67.4

61.8

ELMo+positione

54.2

74.9

62.9

ELMo+position+POSf

56.3

72.7

63.5

BioBERT+position+POSg

57.9

70.1

63.4

The input representation of the model is the word embedding, which is pretrained by word2vec.

b

The input representation of the model is the concatenation of the word embedding and position embedding.

c

The input representation of the model is the concatenation of the word embedding, position embedding, and part of speech (POS) embedding. The
F-measure (%) for this representation was an important finding.
d

The input representation of the model is the deep contextualized word representation.

e

The input representation of the model is the deep contextualized word representation and position embedding.

f

The input representation of the model is the deep contextualized word representation, position embedding, and POS embedding. The F-measure (%)
for this representation was an important finding.
g

The word representation is generated from the last hidden layer of the bidirectional encoder representations from transformers for biomedical text
mining (BioBERT) [33] in a feature-based approach, which means that the parameters of the BioBERT are not fine-tuned. The input representation of
the model is the BioBERT word representation, position embedding, and POS embedding.

In Table 2, we can observe that the model achieves an F-measureof 57.0% when we only use the pretrained word embedding as
the input representation. When we concatenate the pretrained
word embedding and position embedding, the F-measure is
improved from 57.0% to 58.2%, which yields a 1.2%
improvement. When we concatenate the pretrained wordembedding, position embedding, and POS embedding as the
input representations, we yield another 1.9% improvementcompared with only using the pretrained word embedding andposition embedding. The result indicates that both POS andposition features are effective for the CID relation extraction.
The deep contextualized word representation ELMo significantly
outperforms the pretrained word embedding and yields a 4.8%
improvement in the F-measure. The result indicates that ELMo
can generate a more comprehensive representation for the word
according to the sentence context, which results in a better CDR
performance. Similarly, combining the position and POS
embedding with the deep contextualized word representation
can further improve the performance. When we concatenate thedeep contextualized word representation, position embedding,and POS embedding as the input representation, we achieve thebest F-measure of 63.5%. We also use the word representations
generated from the bidirectional encoder representations fromtransformers for biomedical text mining in a feature-based
approach and achieve an F-measure of 63.4%, which is similar
to using ELMo.

Effect of the Attention Mechanism
We evaluated the effectiveness of the multihead self-attention
mechanism. We used the same model architecture that we

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

proposed, but we dealt with the output of BiLSTM by different
attention mechanisms. The attention mechanism is divided into
2 categories: single-head attention mechanism and multihead
attention mechanism. In single-head attention mechanism, we
use 3 types of attention function: additive attention, generalattention, and scaled dot-product attention, as shown below.

(17)

(18)

(19)
where hi is the output of the BiLSTM, W1, W2, s, v are theparameter matrixes, and d is the dimension of the output of the
BiLSTM model. The formula of the multihead attention is
described in formula (11). The comparison performance of thedifferent attention mechanism is presented in Table 3.

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wang et al

Table 3. The effect of the attention mechanism on performance.
Attention mechanism

Precision (%)

Recall (%)

F-measure (%)

Without attention

55.1

71.3

62.2

Additive attention

55.9

70.3

62.3

General attention

55.3

71.8

62.5

Scaled dot-product attention

54.9

73.3

62.8

Multihead attention

56.3

72.7

63.5

In Table 3, we can see that using the attention mechanism can
improve the performance of the CID relation extraction. The
multihead attention mechanism is more helpful than othersingle-head attention mechanisms. This suggests that themultihead attention mechanism can capture more valuable
features from different representation subspaces.

Effect of the Attention Heads
We evaluated the effectiveness of the number of heads of themultihead attention mechanism. In this comparative experiment,
we used the deep contextualized word representation, position
embedding, and POS embedding as the input representation,and the dimensions of query, key, and value are the same. As
shown in Table 4, we only varied the number of heads of themultihead attention.

Table 4. The effect of the attention heads on performance.
Heads (n)Precision (%)

Recall (%)

F-measure (%)

2

57.2

68.2

62.2

4

56.9

70.6

63.0

5

56.3

72.7

63.5

8

57.0

70.2

62.9

10

54.4

75.4

63.2

In Table 4, we can see that the multihead attention mechanism
can effectively improve the performance of the CID relation
extraction. We can observe that the F-measure ranges from
62.2% to 63.5% when setting a different number of heads. Whenthe number of heads is too little or too large, the performance
will drop off. In short, we achieve the best F-measure of 63.5%
when we set the number of heads as 5.

Ablation Study
To examine the contributions of the 2 main components, namely,
multihead attention layer and GCN layer, we ran an ablation
study. The experimental results are shown in Table 5. The results
contain intrasentence level, intersentence level, and relation
merging, which means that merging the intrasentence andintersentence level results in the final document-level result.

Table 5. An ablation study for our model.aModelIntrasentence level
Precision (%)

Relation merging

Recall (%)

F-measure(%)

Precision(%)

Recall (%)

F-measure(%)

Precision (%)

Recall (%)

F-measure(%)

Without multi- 58.2
head attention

82.9

68.4

44.7

44.3

44.5

55.1

71.3

62.2

Without

62.6

74.1

67.9

43.6

48.4

45.9

57.1

66.4

61.4

59.1

81.5

68.5

47.8

52.2

49.9

56.3

72.7

63.5

GCN

b

Our modelaIntersentence level

The values in italics indicate significant findings.

b

GCN: graph convolutional network.

We can observe that removing either the multihead attention
layer or the GCN layer reduces the performance of the model.
This suggests that both layers can learn effective features. When
we remove the multihead attention layer and the GCN layer,
the F-measure drops by 1.3% and 2.1%, respectively. In
particular, we can observe that adding either the multihead
attention layer or the GCN layer improves the performance in
the intersentence level relation extraction by a large margin.
https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

When we remove the multihead attention layer and the GCN
layer, the intersentence level F-measure drops by 5.4% and4.0%, respectively. This suggests that the multihead attention
layer can capture the relatively important features from different
representation subspaces and the GCN layer can capture
long-range syntactic features for intersentence level relation
extraction.
JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 9
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Comparison with Related Work
We compared our model with several state-of-the-art methodsof the CID relation extraction. These methods are divided into
2 categories: methods without additional resources (without
knowledge bases) and methods using additional resources (withknowledge bases). These following methods have beensummarized in Table 6.
1.

2.

3.

Pattern rule-based: Lowe et al [5] developed a pattern-based
system with some heuristic rules to extract CID relations
within the same sentence, and they achieved an F-measureof 60.8%.
Maximum entropy model: Gu et al [7] developed a machinelearning-based system that utilized simple but effective
manual linguistic features with the maximum entropy
model. They built rich manual features for intrasentencelevel and intersentence level instances. They achieved an
F-measure of 58.3%.
LSTM+ support vector machine (SVM): Zhou et al [10]developed a hybrid system, which consists of afeature-based model that utilized flat features and structurefeatures with SVM and a neural network model based on
LSTM. Their model achieved an F-measure of 56.0%. Afterusing additional postprocessing heuristic rules, they
achieved a 5.3% improvement in the F-measure.

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

Wang et al
4.

5.

6.

7.

8.

9.

CNN+maximum entropy: Gu et al [11] proposed amaximum entropy model for intersentence level relation
extraction and a CNN model for intrasentence level relation
extraction. They achieved an F-measure of 60.2%. They
also used additional postprocessing heuristic rules toimprove performance that increases the F-measure to 61.3%.
Biaffine Relation Attention Network: Verga et al [12]proposed this based on the multihead self-attention model,
which can predict relationships between all the mentioned
pairs in the document. The model achieved an F-measureof 62.1%.
Graph convolutional neural network: Sahu et al [18]proposed a labelled edge graph convolutional neuralnetwork model on a document-level graph. The model
achieved an F-measure of 58.6%.
SVM_Xu: Xu et al [6] explored 4 different knowledge bases
to extract the knowledge features and achieved an
F-measure of 67.2%.
SVM_Pons: Pons et al [9] extracted 3 sets of features, which
are prior knowledge and statistical and linguistic
information from the document. They achieved an
F-measure of 70.2%.
Knowledge-guided convolutional network: Zhou et al [34]proposed a CNN that integrated both relation representations
and entity representations learned from knowledge bases.
The model achieved an F-measure of 71.3%.

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 10(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wang et al

Table 6. Comparisons with related work.
Category and method

Precision (%)

Recall (%)

F-measure (%)

59.3

62.3

60.8

62.0

55.1

58.3

LSTM+SVMb

64.9

49.3

56.0

LSTM+SVM+PPc

55.6

68.4

61.3

CNN+MEd

60.9

59.5

60.2

CNN+ME+PP

55.7

68.1

61.3

55.6

70.8

62.1

52.8

66.0

58.6

56.3

72.7

63.5

65.8

68.6

67.2

73.1

67.6

70.2

69.7

72.9

71.3

Without knowledge bases
Lowe et al [5]Pattern rule-based
Gu et al [7]MEaZhou et al [10]Gu et al [11]Verga et al [12]BRANe
Sahu et al [18]GCNNf
Our study
GCNg+Multihead attention
With knowledge bases
Xu et al [6]SVM
Pons et al [9]SVM
Zhou et al [34]KCNha

ME: maximum entropy model.

b

LSTM+SVM: long short-term memory+support vector machine.

c

LSTM+SVM+PP: long short-term memory+support vector machine+postprocessing.

d

CNN+ME: convolutional neural network+maximum entropy model.

e

BRAN: biaffine relation attention network.

f

GCNN: graph convolutional neural network.

g

GCN: graph convolutional network.

h

KCN: knowledge-guided convolutional networks.

In Table 6, the deep neural network-based methods achieved
competitive performance in the CID relation extraction task.
For example, Sahu et al [18] used GCN to capture dependency
information and achieved an F-measure of 58.6%. Comparedwith other deep neural network-based methods, we not only
employed the multihead attention to capture the relatively
important semantic features but also used the GCN to capture
the valuable syntactic features from the document-level
dependency graph automatically and effectively. We also
observed that some studies [7,10,11] designed and extractedrich semantic and syntactic features for the relation extraction
task and used additional postprocessing heuristic rules tohttps://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

improve performance. Our method is an end-to-end neuralnetwork-based model and achieves a high F-measure of 63.5%
without using postprocessing heuristic rules. As shown in Table
6, the methods with knowledge bases outperform the methods
without knowledge bases significantly. This suggests that priorknowledge is much useful for CID relation extraction. In thisstudy, we focus on the effectiveness of GCN and multihead
attention mechanism rather than the prior knowledge. We willattempt to integrate the biomedical knowledge to further
improve the performance of our method in our future work.

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 11(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Visualization of Multihead Attention Mechanisms
To understand our multihead self-attention mechanism clearly,
we visualized the attention weights of an example sequence inFigure 4. Different colors represent different heads. The darker
the color is, the higher the attention weight is. In Figure 4, theword pays different levels of attention to different words in
different heads. For the word “Cardiac,” the word “Pilsicainide”

Wang et al
has the higher weight score in the second head; however, thewords “Torsades” and “Pointes” have the higher weight scorein the last head. For the word “Pilsicainide,” the words“Cardiac” and “Death” have the higher weight score in the third
head; however, the word “Torsades” has the higher weight scorein the fourth head. Thus, the multihead self-attention mechanism
can make the model capture the relatively important featuresfrom different representation subspaces.

Figure 4. Examples of the multi-head self-attention mechanism. Attentions here shown only for the words "Cardiac" and "Pilsicainide." Different
colors represent different heads.

Error Analysis
To understand our model better, we performed an error analysis
on the output of our final results. There are the 2 main types oferrors: false positive errors and false negative errors. We list
some examples to analyze the errors. In false positive errors,some instances are nonrelations but are mistaken as CID
relations. For the sentences “Carbamazepine (Chemical:
D002220)-induced cardiac dysfunction (Disease: D006331)”and “A patient with sinus bradycardia and atrioventricular block(Disease: D054537) induced by carbamazepine (Chemical:
D002220),” the disease D006331 is the hypernym of the disease
D054537. According to the labeling rules of the CDR corpus,
we need to extract the most specific relations. Thus, the first
sentence does not express a CID relation and the second
sentence expresses a CID relation. However, our model extracts
a CID relation between the chemical D002220 and the disease
D006331 in the first sentence incorrectly because the first
sentence is the common sentence pattern that expresses a CID
relation. In false negative errors, several CID relations are notrecognized. One of the main reasons for some intersentencelevel instances to be removed by the heuristic rules in therelation instance construction stage is because the sentence
distance is more than 3. In the future, we will consider preferable
preprocessing and postprocessing techniques to solve the above
problems.

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

Discussion
In this paper, we propose a novel end-to-end neural network
based on GCN and multihead attention. The document-level
dependency graph is constructed to capture the dependency
syntactic information across sentences. We applied GCN to
capture the long-range dependency syntactic features, which
can improve the performance of intersentence level relation
extraction. Further, we employed the multihead attention
mechanism to capture the relatively important context featuresfrom different semantic subspaces. ELMo is used in our model
to enhance the input representation. We evaluate theeffectiveness of ELMo, multihead attention mechanism, and
GCN on the BioCreative V CDR dataset. Experimental results
show that ELMo, multihead attention, and GCN cansignificantly improve the performance of the CDR extraction.
Our method achieves an F-measure of 63.5%, which is superior
to other state-of-the-art methods. There are many large-scale
knowledge bases such as the Comparative Toxicogenomics
Database, Unified Medical Language System, Medical Subject
Headings, UniProt, and the commercial system Euretos
Knowledge Platform. These knowledge bases contain a large
amount of structured data in the form of triples (entity, relation,
entity), wherein relation represents the relationship between 2
entities. Some studies suggest that integrating the structured
information from the knowledge bases may improve theperformance of the CDR extraction. In future studies, we will

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 12(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
integrate the biomedical knowledge to further improve theWang et al
performance of our method.

Acknowledgments
The work was supported by grants from National Natural Science Foundation of China (No. 61572098 and 61572102). We would
like to thank the Natural Science Foundation of China. We also would like to thank all the anonymous reviewers for their valuablesuggestions and constructive comments.

Authors' Contributions
JW and YZ led the method application, experiment conduction, and the result analysis. XC, YZ, and JW participated in the data
extraction and preprocessing. YZ and XW participated in the manuscript revision. HL and ZY provided theoretical guidance andthe revision of this paper.

Conflicts of Interest
None declared.

References
1.
2.
3.
4.

5.

6.
7.
8.

9.

10.

11.
12.

13.

14.

15.

16.

Islamaj Dogan R, Murray GC, Névéol A, Lu Z. Understanding PubMed user search behavior through log analysis. Database(Oxford) 2009 Nov 27;2009:bap018-bap018 [FREE Full text] [doi: 10.1093/database/bap018] [Medline: 20157491]
Hurle MR, Yang L, Xie Q, Rajpal DK, Sanseau P, Agarwal P. Computational drug repositioning: from data to therapeutics.
Clin Pharmacol Ther 2013 Apr 15;93(4):335-341. [doi: 10.1038/clpt.2013.1] [Medline: 23443757]
Qu J, Ouyang D, Hua W, Ye Y, Li X. Distant supervision for neural relation extraction integrated with word attention andproperty features. Neural Netw 2018 Apr;100:59-69. [doi: 10.1016/j.neunet.2018.01.006] [Medline: 29471196]
Wei C, Peng Y, Leaman R, Davis AP, Mattingly CJ, Li J, et al. Assessing the state of the art in biomedical relation extraction:
overview of the BioCreative V chemical-disease relation (CDR) task. Database (Oxford) 2016;2016:baw032 [FREE Fulltext] [doi: 10.1093/database/baw032] [Medline: 26994911]
Lowe DM, O'Boyle NM, Sayle RA. Efficient chemical-disease identification and relationship extraction using Wikipedia
to improve recall. Database (Oxford) 2016 Apr 08;2016:baw039 [FREE Full text] [doi: 10.1093/database/baw039] [Medline:
27060160]
Xu J, Wu Y, Zhang Y, Wang J, Lee HJ, Xu H. CD-REST: a system for extracting chemical-induced disease relation inliterature. Database (Oxford) 2016;2016:baw036 [FREE Full text] [doi: 10.1093/database/baw036] [Medline: 27016700]
Gu J, Qian L, Zhou G. Chemical-induced disease relation extraction with various linguistic features. Database (Oxford)
2016 Apr 06;2016:baw042 [FREE Full text] [doi: 10.1093/database/baw042] [Medline: 27052618]
Zhou H, Deng H, He J. Chemical-disease relations extraction based on the shortest dependency path tree. 2015 Presented
at: Proceedings of the Fifth BioCreative Challenge Evaluation Workshop; 2015; Seville, Spain p. 214-219 URL: https://pdfs.semanticscholar.org/e66a/754947a9abd6665ab16815f52bc1c9aed596.pdf
Pons E, Becker BF, Akhondi SA, Afzal Z, van Mulligen EM, Kors JA. Extraction of chemical-induced diseases using priorknowledge and textual information. Database (Oxford) 2016 Apr 14;2016:baw046 [FREE Full text] [doi:10.1093/database/baw046] [Medline: 27081155]
Zhou H, Deng H, Chen L, Yang Y, Jia C, Huang D. Exploiting syntactic and semantics information for chemical-disease
relation extraction. Database (Oxford) 2016 Apr 14;2016:baw048 [FREE Full text] [doi: 10.1093/database/baw048][Medline: 27081156]
Gu J, Sun F, Qian L, Zhou G. Chemical-induced disease relation extraction via convolutional neural network. Database(Oxford) 2017 Jan 01;2017(1):bax024 [FREE Full text] [doi: 10.1093/database/bax024] [Medline: 28415073]
Verga P, Strubell E, Mccallum A. Simultaneously self-attending to all mentions for full-abstract biological relation extraction.
2018 Presented at: the 16th Annual Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies; 2018; New Orleans, USA. [doi: 10.18653/v1/N18-1080]
Nguyen D, Verspoor K. Convolutional neural networks for chemical-disease relation extraction are improved withcharacter-based word embeddings. 2018 Presented at: Proceedings of the BioNLP workshop; 2018; Melbourne, Australia
p. 129-136. [doi: 10.18653/v1/W18-2314]
Peters ME, Neumann M, Iyyer M. Deep contextualized word representations. 2018 Presented at: the 16th Annual Conferenceof the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; 2018;
New Orleans, USA p. 2227-2237. [doi: 10.18653/v1/N18-1202]
Li L, Nie Y, Han W, Huang J. A Multi-attention-Based Bidirectional Long Short-Term Memory Network for Relation
Extraction. 2017 Presented at: International Conference on Neural Information Processing; 2017; Guangzhou, China p.
216-227. [doi: 10.1007/978-3-319-70139-4_22]
Vaswani A, Shazeer N, Parmar N. Attention is all you need. 2017 Presented at: Neural Information Processing Systems(NIPS);
2017; Long Beach, USA p. 5998-6008 URL: https://dl.acm.org/doi/pdf/10.5555/3295222.3295349

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 13(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
17.

18.

19.

20.
21.
22.

23.

24.
25.

26.
27.

28.

29.
30.
31.
32.
33.

34.

Wang et al

Quirk C, Poon H. Distant supervision for relation extraction beyond the sentence boundary. 2017 Presented at: the 15th
Conference of the European Chapter of the Association for Computational Linguistics; 2017; Valencia, Spain p. 1171-1182.
[doi: 10.18653/v1/e17-1110]
Sahu SK, Christopoulou F, Miwa M, Ananiadou S. Inter-sentence Relation Extraction with Document-level Graph
Convolutional Neural Network. 2019 Presented at: the 57th Annual Meeting of the Association for Computational Linguistics;
2019; Florence, Italy p. 4309-4316. [doi: 10.18653/v1/p19-1423]
Krallinger M, Vazquez M, Leitner F, Salgado D, Chatr-Aryamontri A, Winter A, et al. The Protein-Protein Interaction
tasks of BioCreative III: classification/ranking of articles and linking bio-ontology concepts to full text. BMC Bioinformatics
2011 Oct 03;12 Suppl 8:S3 [FREE Full text] [doi: 10.1186/1471-2105-12-S8-S3] [Medline: 22151929]
Segura-Bedmar I, Martínez P, Herrero-Zazo M. Lessons learnt from the DDIExtraction-2013 Shared Task. J Biomed Inform
2014 Oct;51:152-164 [FREE Full text] [doi: 10.1016/j.jbi.2014.05.007] [Medline: 24858490]
Coletti MH, Bleich HL. Medical subject headings used to search the biomedical literature. J Am Med Inform Assoc
2001;8(4):317-323 [FREE Full text] [doi: 10.1136/jamia.2001.0080317] [Medline: 11418538]
Zhao Z, Yang Z, Luo L, Lin H, Wang J. Drug drug interaction extraction from biomedical literature using syntax convolutional
neural network. Bioinformatics 2016 Nov 15;32(22):3444-3453 [FREE Full text] [doi: 10.1093/bioinformatics/btw486][Medline: 27466626]
Zhang Y, Zheng W, Lin H, Wang J, Yang Z, Dumontier M. Drug-drug interaction extraction via hierarchical RNNs on
sequence and shortest dependency paths. Bioinformatics 2018 Mar 01;34(5):828-835 [FREE Full text] [doi:10.1093/bioinformatics/btx659] [Medline: 29077847]
Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput 1997 Nov 15;9(8):1735-1780. [doi:10.1162/neco.1997.9.8.1735] [Medline: 9377276]
Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate. 2015 Presented
at: 3rd International Conference on Learning Representations; May 7-9, 2015; San Diego, CA, USA URL: https://arxiv.
org/pdf/1409.0473.pdf
Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc. IEEE 1998
Nov;86(11):2278-2324. [doi: 10.1109/5.726791]
Zhang Y, Qi P, Manning CD. Graph Convolution over Pruned Dependency Trees Improves Relation Extraction. 2018
Presented at: Conference on Empirical Methods in Natural Language Processing; 2018; Brussels, Belgium p. 2205-2215.
[doi: 10.18653/v1/d18-1244]
Santoro A, Raposo D, Barrett D, Malinowski M, Pascanu R, Battaglia P, et al. A simple neural network module for relational
reasoning. 2017 Presented at: Advances in Neural Information Processing Systems; 2017; Long Beach, USA p. 4967-4976
URL: https://papers.nips.cc/paper/7082-a-simple-neural-network-module-for-relational-reasoning.pdf
Lee K, He L, Lewis M, Zettlemoyer L. End-to-end neural coreference resolution. 2017 Presented at: the 2017 Conferenceon Empirical Methods in Natural Language Processing; 2017; Copenhagen, Denmark p. 188-197. [doi: 10.18653/v1/d17-1018]
Kingma DP, Ba J. Adam:A Method for Stochastic Optimization. 2015 Presented at: the 3rd International Conference forLearning Representations; 2015; San Diego URL: https://arxiv.org/pdf/1412.6980.pdf
An open source machine learning framework that accelerates the path from research prototyping to production deployment.
PyTorch: From Research to Production. URL: https://pytorch.org/ [accessed 2020-04-08]
Software. The Stanford Natural Language Processing Group. URL: https://nlp.stanford.edu/software/ [accessed 2020-05-04]
Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model
for biomedical text mining. Bioinformatics 2020 Feb 15;36(4):1234-1240. [doi: 10.1093/bioinformatics/btz682] [Medline:
31501885]
Zhou H, Lang C, Liu Z, Ning S, Lin Y, Du L. Knowledge-guided convolutional networks for chemical-disease relation
extraction. BMC Bioinformatics 2019 May 21;20(1):260 [FREE Full text] [doi: 10.1186/s12859-019-2873-7] [Medline:
31113357]

Abbreviations
BiLSTM: bidirectional long short-term memory
CDR: chemical-disease relation
CID: chemical-induced disease
CNN: convolutional neural network
GCN: graph convolutional network
LSTM: long short-term memory
NLP: natural language processing
POS: part of speech
SVM: support vector machinehttps://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 14(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wang et al

Edited by T Hao, B Tang, Z Huang; submitted 30.12.19; peer-reviewed by I Gabashvili, L Li; comments to author 01.03.20; revised
version received 14.04.20; accepted 25.04.20; published 31.07.20
Please cite as:
Wang J, Chen X, Zhang Y, Zhang Y, Wen J, Lin H, Yang Z, Wang X
Document-Level Biomedical Relation Extraction Using Graph Convolutional Network and Multihead Attention: Algorithm Development
and Validation
JMIR Med Inform 2020;8(7):e17638
URL: https://medinform.jmir.org/2020/7/e17638
doi: 10.2196/17638
PMID: 32459636©Jian Wang, Xiaoyu Chen, Yu Zhang, Yijia Zhang, Jiabin Wen, Hongfei Lin, Zhihao Yang, Xin Wang. Originally published in
JMIR Medical Informatics (http://medinform.jmir.org), 31.07.2020. This is an open-access article distributed under the terms of
the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly
cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as thiscopyright and license information must be included.

https://medinform.jmir.org/2020/7/e17638

XSL• FO
RenderX

JMIR Med Inform 2020 | vol. 8 | iss. 7 | e17638 | p. 15(page number not for citation purposes)