Learning to Leverage High-Order Medical Knowledge Graph for Joint Entity and Relation Extraction
Year: 2023

Abstract:
Automatic medical entity and relation extraction is essential for daily electronic medical record (EMR) analysis, and has attracted a lot of academic attention. Tremendous progress has been made in recent years. However, medical terms are difficult to understand, and their relations are more complicated than general ones. Based on this situation, domain knowledge gives better background and contexts for medical terms. Despite the benefits of medical domain knowledge, the utilization way of it for joint entity and relation extraction is inadequate. To foster this line of research, in this work, we propose to leverage the medical knowledge graph for extracting entities and relations for Chinese Medical Texts in a collective way. Specifically, we propose to construct a high-order heterogeneous graph based on medical knowledge graph, which is linked to the entity mentions in the text. In this way, neighbors from the high-order heterogeneous graph can pass the message to each other for better global context representations. Our experiments on real Chinese Medical Texts show that our method is more effective than state-ofthe-art methods.

heading: 1 Introduction
text: Medical text, e.g., electronic medical record (EMR), has been produced at a rapid speed and massive volume every day. Without any structured organization, this enormous volume of medical information is difficult to be read through by humans in a short time (Shang et al., 2021). Due to this situation, many researchers have recently paid great attention to the joint entity and relation extraction in the medical domain (Lai et al., 2021; Verlinden et al., 2021).
The challenge of joint entity and relation extraction in medical domain is that medical terms are
∗Corresponding authors
usually difficult to understand due to the requirement of medical domain knowledge, especially for abbreviations of medical terms in the medical text. Even worse, relations between medical entities become even more complicated. Therefore, medical domain knowledge that could provide meaningful contexts and backgrounds is essential for the better extraction of medical entities and relations. Despite the advantages of medical domain knowledge, most previous works fail to use medical domain knowledge (Li et al., 2017; Xue et al., 2019; Pang et al., 2021; Luo et al., 2020). They solely rely on the local information in the medical text to extract entities and relations with language model (LM), which is insufficient for incomprehensible medical terms and complicated relations between entities.
Some recent works utilize medical knowledge for joint entity and relation extraction (Lai et al., 2021; Verlinden et al., 2021). However, both Lai et al. (2021) and Verlinden et al. (2021) simply align entity representation (node representation) from knowledge graph to local texts and fail to explicitly introduce the complicated relation contexts (edge representation) in the medical knowledge graph to enhance the deep representations of their involved entities. Huang et al. (2020) propose graph edge-conditioned attention network (GEANet) which integrates initial static relation embedding into attention mechanism for entity representation enhancement in medical knowledge graph. Nonetheless, it leaves out relational update during knowledge graph training process. Battaglia et al. (2018) propound graph network (GN) framework to update node and edge features iteratively within a heterogeneous graph. However, the edge representation update is based on the sender and receiver nodes information it links, which will bring about fluctuation since the amount of nodes is far outweighed that of edge types.
Therefore, we propose a method to fix these issues by providing additional relation contexts from
9023
medical knowledge graph to enrich the deep representations of entity mentions. Specifically, we propose to construct a high-order heterogeneous graph, e.g., Fig. 1, to provide meaningful global contexts for its linked entity mentions. In Fig. 1, we denote Ge as the standard first-order graph with entities as nodes and relations as edges, and represent Gr as Ge’s converted second-order graph with relations as nodes and entity types as edges. For every relation pair of an entity in Ge, e.g., r3 and r2, we link an edge, i.e., the entity type te1 , in Gr connecting them. In this way, both message passing of entities via different relations and message propagation of relations via different entity types can be well diffused in the global graph structure. After extracting the high-order heterogeneous graph from the medical knowledge graph, we fuse the entity and relation representations in the global context obtained from the high-order heterogeneous graph with the local information extracted from the medical text.
To summarize, our contributions are: • We propose a high-order graph modeling method for knowledge fusion, which treats text related sub-graph as the first-order graph with entities as nodes and its converted graph as the secondorder one with relations as nodes. We update the hidden representations of nodes in the two order graphs separately as the entity/relation representations for knowledge graph. • We present a knowledge-enhancement method for medical text encoding, which boosts the entity representation of the first-order graph with the feedback of the second-order relation representation. And it further enhances the encoding of the entity mentions from medical text for joint extraction. • We have performed substantial experiments against existing methods. Our evaluation results on real medical datasets verify that our method is more effective than state-of-the-art methods.
The rest of this paper is organized as follows.
Section 2 discusses related work. In Section 3, we introduce the typical algorithms for text and graph representation and present the proposed method for knowledge-enhanced joint extraction. Section 4 shows the evaluation results on two datasets with compared to some other advanced methods. We conclude in Section 5.

heading: 2 Related Work
text: There are two categories of entity and relation extraction methods: pipeline-based methods and joint extraction methods.
Pipeline-based entity and relation extraction methods: These work usually first extract entities as outputs, then extract relations for the returned entities. The drawback of pipeline-based extraction methods is that the errors of entity extraction may be accumulated when extracting relations for the already returned entities. For example, Zhong and Chen (2021) put forward an extra encoder and fuse the entity type information to enhance entity pair representation during the relation extraction task.
Jointly entity and relation extraction methods: Some recent work extracts entity and relation in a collective way to overcome the accumulated error problem in the pipeline-based methods. There are two kinds of standard extraction methods, which are non-knowledge-enhanced and knowledge-enhanced methods.
Most joint entity and relation extraction methods ignore the domain knowledge. Wang et al. (2018) utilize a novel graph scheme to solve the problem. Luan et al. (2018) apply multi-task method to optimize entities, relations, and coreference simultaneously.Bekoulis et al. (2018) further propose multi-context based adversarial training method. Luan et al. (2019) utilize dynamic span graphs to form a general framework. Fu et al. (2019) model text as relational graphs for joint extraction. Zhao et al. (2021) model dense cross-modal interactions for joint extraction. Wang and Lu (2020) apply table-sequence encoders to extract jointly. Lin et al. (2020) apply neural model for information extraction with global features. Recently, Eberts and Ulges (2020) propose a span-based method with transformer pre-training. Further, Ji et al. (2020) apply span-based method with attention-based spanspecific and contextual semantic representations. Moreover, Wei et al. (2020) propose a cascade binary tagging framework for entity and relation extraction. Yan et al. (2021) propose a partition filter
network for joint entity and relation extraction. Despite that these works extract entity and relation in a joint way, they are not designed for medical text. While some previous works are designed for medical text, they ignore the help of medical domain knowledge when modeling (e.g., Li et al., 2017; Xue et al., 2019; Pang et al., 2021; Luo et al., 2020). Only a few works incorporate medical domain knowledge to enhance the contexts of the medical text for joint entity and relation extraction (Lai et al., 2021; Verlinden et al., 2021). Among them, Lai et al., 2021 extract entity and relation jointly with knowledge-enhanced collective inference. Verlinden et al. (2021) inject knowledge base information into entity and relation extraction and coreference resolution simultaneously. However, both Lai et al. (2021) and Verlinden et al. (2021) fail to explicitly incorporate complicated relation contexts for their involved entities. Thus the interaction between entity and relation extraction will not be captured. Our method overcomes this drawback by providing relation contexts when modeling deep representations of entities.
Most of the current work does not use domain knowledge, or only uses the entity context in domain knowledge, and does not include the relation context. While we explicitly model the entity context and relation context as the important context information for entity span, which improves the results of joint extraction.

heading: 3 Proposed Method
text: Fig. 2 shows the framework of our method. It first extracts the high-order heterogeneous graph from the medical knowledge graph (Section 3.1), and learns entity and relation representation from the global context (Section 3.2). After that, we learn the representation of entity mentions from the local context in the medical text (Section 3.3). The fusion of entity representations from both global and local contexts and its corresponding relation representation (Section 3.4) is utilized to extract entities and relations in a collective way (Section 3.5).

heading: 3.1 High-order Heterogeneous Graph Extraction
text: The challenge is that there are multiple relations in the heterogeneous graph, and modeling entities together with their contextual relation information is non-trivial. We propose to construct a high-order
heterogeneous graph from the knowledge graph for the medical text, such that the text representation contains additional global knowledge contexts of related entities and relations.
The high-order heterogeneous graph comprises of first-order graph (a text related sub-graph of original knowledge graph) and its converted secondorder graph. Given a piece of medical text expressed as words t = (w0, w1, ..., wi, ..., wk) and a domain knowledge graph G = (V, ε, E), the goal of high-order heterogeneous graph extraction is to obtain Ge = (Ve, Ee) and Gr = (Vr, Er). Specifically,
Ve = V ∩ {wi}, (ei, rei,ej , ej) ∈ ε, (1) rei,ej = Ee(ei, ej),
where vei , vej ∈ Ve, rei,ej ∈ Ee and ε is the set of all triplets {(eh, reh,et , et)} in knowledge graph. And
Vr ⊆ Ee, Er ⊆ T (Ve), (2)
I(Er(Ee(ei, ej), Ee(el, em)))=I(ei=em∨ej=el),
where T (.) means the type of an entity. I is the indicator function and I(Er(.)) measures whether the edge Er(.) in second-order graph exists.
Hence, it first searches through the medical knowledge graph to extract all triplets as the firstorder graph pertaining to current medical text, and converts it to the second-order graph with nodes and edges switched. To be detailed, we traverse nodes, i.e., entities, in the first-order graph and collect their in and out edges, i.e., relations. Concretely, when it is the head entity in a triplet, we call the corresponding relation the out relation, conversely, the in relation. For any in-out relation pair of an entity, we consider relations as nodes for the second-order graph, and the type of the corresponding entity as the edge linking them. As it can be seen in the High-Order Heterogeneous Graph Extraction Module part of Fig. 2, a Chinese entity “鹅口疮(thrush) ” is revolved around by relations of “可能疾病(possible disease) ”, “并发 症(complication) ” and “传染方式(mode of infection) ” in the first-order graph, during which the first two are in relations, and the other is the out relation. For any in-out relation pair, e.g., “并发 症-传染方式”, we take them as two nodes in the
converted second-order graph and the type of the entity “鹅口疮”, i.e., “疾病(disease) ” as the linking edge. As a counter example, “可能疾病” and “并发症” are all in relations for entity “鹅口疮”, therefore, no edge exists to link them. We merely link the in-out relation pair on account of the message flowing direction, i.e., from head entity to tail entity.
This high-order heterogeneous graph we propose is distinguished from the node layer and edge layer in Jiang et al. (2020), which lies in two aspects:
(1) We only consider the in-out relation pairs as edges for second-order graph, which better reflects the direction of information flow, while Jiang et al. (2020) links all relation pairs.
(2) The edge in second-order graph represents the entity type in our method, however, it remains as the entity by authors of Jiang et al. (2020), where the number of entities is very huge and will not be feasible for GNN to learn.

heading: 3.2 High-Order Heterogeneous Graph Modeling
text: Our idea is to propagate messages among both the first-order graph Ge and the converted secondorder graph Gr in Fig. 2, in order to capture the complex global information from knowledge graph. Hence, entity mentions in the medical text can integrate their local information with the related global
contexts for the better encoding. We first propagate message among the standard first-order graph Ge with relational graph convolutional network (RGCN) (Schlichtkrull et al., 2018). We apply TransE (Bordes et al., 2013) as the initialization of the embedding for each node vi. The embedding of each node vi, i.e., entity, can be updated as:
vl+1i = ReLU(U lvli +
∑
k∈R
∑
vj∈Nki
( 1
|Nki | Ulkv l j)) ,
(3) where vli is the embedding of the node vi at layer l. Nki is the set of neighbors of vi under relation k. Ul is the trainable parameter at layer l. Ulk is relation specific weighted parameter.
We further model the global contexts in the converted second-order graph Gr. Different from the first-order graph Ge, the nodes and edges in secondorder graph represent relations and entity types respectively. We pass messages among the neighbors of the node, i.e., relation ri, via different entity types in graph Gr. Then the information among multiple entity types of relation ri can be summarized. Here the initialization of the embedding for each relation ri is also obtained by the TransE model. The embedding of each node, i.e., relation ri, is integrated by the deep representations of its
neighbors as:
rl+1i = ReLU(O lrli+
∑
t∈T
∑
rj∈Nti
( 1
|N ti | Oltr l j)) , (4)
where rli is the embedding of the relation ri at layer l. N ti is the set of neighbors of relation ri under the entity type t ∈ T , while |N ti | is the number of neighbors of ri. Ol is the trainable parameter at layer l. And ReLU is the activation function.
The high-order heterogeneous graph modeling provides deep representations for the entities and relations. Therefore, complicated relation information can be well preserved for the involved entities in the modeling of Section 3.5.

heading: 3.3 Spans Representation with Transformer Encoder
text: After modeling the high-order heterogeneous graph extracted from the medical knowledge graph, we then model the local contexts in the medical text. The medical text is organized here as tokens t = (x0, x1, ..., xi, ..., xn−1), and a successive sequence (xsi , ..., xei) in text is a span which means an entity mention for medical text. Each span si is modeled as:
si = gs([xsi , xei , x̂i, ϕ(si)]) , (5)
where xsi denotes the token level embedding from the transformer encoder, e.g., BERT, of the start of span si, while xei represents the token level embedding of the end of span si. x̂i is an attentionweighted sum of the token representations in the span. And ϕ(si) is a feature vector modeling the length of si. gs is a feed-forward neural network (Lee et al., 2017).
Then a span-based GCN is applied on the graph with spans as nodes and relations between spans as edges:
hl+1i = h l i + f l span(ReLU(f l s(h l i), f l s′(h l i))) , (6)
and
f ls(h l i) =
∑
sj∈sset,j ̸=i
∑ k∈R rij [k](Wkh l j + bk),
f ls′(h l i) =
∑
sj∈sset,j ̸=i
∑ k∈R rji[k](W ′ kh l j + b ′ k), (7)
rij [k] = Softmax(fr([si, sj , si ◦ sj ]))[k],
where hl+1i is the deep representation of si at the layer l+1, fspanand fr are feedforward neural networks. f ls and f l s′ are bidirectional GCN (Marcheggiani and Titov, 2017; Fu et al., 2019) for hli, rij [k]
measures the relation score for relation k between spans of si and sj , ◦ denotes the element-wise multiplication. And h0i is initialized as si.

heading: 3.4 Knowledge Graph Enhanced Span Representation
text: After applying span-based GCN, we obtain the hidden representation hi of span si derived from local context, e.g., the medical text. Then we apply an attention mechanism (Lai et al., 2021) to integrate the deep representation of hidden representation hi of span si from the local medical text with the deep representation of entities vi from the global contexts in the first-order graph as fie :
fie = Wiefce(hi) + ∑
vj∈C(sie ) Wijefv(vj) , (8)
where C(sie) is the candidate set of entities corresponding to span si in the dual heterogeneous graph. And fce(hi) and fv(vj) are the transformed representation of hi and vj by two feedforward neural networks. And Wie and Wije are attention scores of the two transformed representations:
Wie = exp(αie)
(exp(αie) + ∑ vj∈C(si) exp(αije)) , (9)
and
Wije = exp(αije)
(exp(αie) + ∑ vj∈C(si) exp(αije)) .
(10) Here αije and αie are importance scores of the transformed entity representation vj and the transformed span representation hi to the span representation hi:
αije = fαe([hi, fv(vj)]) , (11)
and αie = fαe([hi, fce(hi)]) , (12)
where fαe is a feedforward neural network. Next, we fuse the deep representations of relations ri from the global contexts in the secondorder graph Gr with the deep representation hi of span si from the local medical text. Distinguished from first-order graph fusion in equation 8, we argue that the corresponding relations in secondorder graph may be irrelevant to the span depending on current medical text, and arouse disproportionate or even noisy aggregating representations for
local context. Therefore, a selective gate mechanism (Li et al., 2020) is utilized to perform this fusion as fir :
fir = girfcr(hi) + ∑
rj∈C(sir ) gijrfr(rj), (13)
where C(sir) is the candidate set of relations in the second-order heterogeneous graph for the span si. To be detailed, C(sir) is a subset of relations from knowledge graph triplets where si holds as head or tail entity. And fcr(hi) and fr(rj) are the transformed representation of hi and rj by two feedforward neural networks. And gir and gijr are gate scores of the two transformed representations:
gir=σ(W2(ReLU(W1[fcr(hi), hi]+b1)+b2), (14)
and
gijr=σ(W2(ReLU(W1[fr(rj), hi]+b1)+b2), (15)
where σ is the Sigmoid activate function which maps the results into the interval of (0, 1).
Then, the fused representation fi can be modeled as:
fi = (Wsumefie +Wsumrfir)||hi , (16)
which means an integration of deep representation among entities, relations and the span. Wsume and Wsumr serve as feedforward neural networks. || means concatenation operation.

heading: 3.5 Collective Entity and Relation Extraction
text: Finally, we map the integrated representation fi to the entity type space as:
ei = Softmax(ge(fi)) , (17)
where ge is a feed-forward neural network to map fi to the entity space.
Similarly, the relation between span i and j can be mapped to the relation type space as:
rij = Softmax(gr(fi, fj)) , (18)
where gr is a feed-forward neural network to map fi and fj to the relation space.
The training of collectively extracting entity and relation can be optimized by minimizing the loss function as:
L = Le + Lr , (19)
where Le represents the cross-entropy loss of entities, and Lr denotes the cross-entropy loss of relations.
Different from the pipeline way that optimizes entities and relations in separate steps, our method conducts the optimization in an end-to-end mode. In this way, the errors of extracting entities and relations can be reduced collectively.

heading: 4 Experiments
text: In this section, we evaluate our model with extensive experiments. We first show the experimental setup, which contains datasets, baselines for comparison, and evaluation metrics. Then we evaluate the effectiveness of our model and baselines.

heading: 4.1 Experimental Setup

heading: 4.1.1 Datasets
text: We evaluate our method on several real medical datasets with one medical knowledge graph dataset.
Chinese Medical Text datasets: We evaluate our model on three Chinese medical text datasets. • The first medical text dataset is the CHIP2020 dataset 1, which contains 17,924 sentences from biomedical Chinese text that captures relations between medical entities (Guan et al., 2020). The dataset includes a pediatric labeled corpus for hundreds of common diseases. The training and testing splits contain 14,339 and 3,585 sentences, respectively. • The next medical dataset is the CHIP-2022 dataset 2, which contains 1000 samples and is split into 850/150 for training and testing set, respectively. CHIP2022 aims to extract casual, entailment and conditional relations between entities whose type are not distinguished. In the experiments, we merely trace out the previous two types of relations for joint extraction. • The third medical text dataset is the DiaKG dataset 3, which contains sentences from diabetes text that reflects relations between medical entities (Chang et al., 2021). The dataset comes from 41 diabetes guidelines and consensuses from authoritative journals in China. The diabetes text contains 22,050 entities and 6,890 relationships. Finally, the prepossessed training and testing splits contain 1,170 and 239 sentences, respectively.
1http://cips-chip.org.cn/2020/eval2 2http://cips-chip.org.cn/2022/eval2 3https://tianchi.aliyun.com/dataset/
dataDetail?dataId=88836
Chinese Medical knowledge graph dataset: We extract Chinese medical triplets from the public knowledge graph dataset 4, which contains diseaserelated entities and disease-related triples. After preprocessing, we obtain 215,745 triplets for the medical knowledge graph. These triplets contain 16,735 entities and 13 relations, and the amount of entity types is 9.

heading: 4.1.2 Baselines for comparison
text: We compare our model with state-of-the-art joint extraction methods, variant versions of knowledgeenhanced baseline included. • PFN (Partition Filter Network) is an advanced joint extraction method that does not utilize a knowledge graph. It segments the encoder into entity extraction and relation extraction parts, and accomplishes NER-specific and relation-specific tasks with shared part separately (Yan et al., 2021). • CASREL (Cascade Binary Tagging Framework for Relational Triple Extraction) is an advanced joint extraction method without knowledge graph enhancement. It models triplets extraction into head entity extraction as well as relation and tail entity extraction with fused head entity representation. The two parts share the same encoder for a joint extraction task (Wei et al., 2020). • SPERT (Span-based Joint Entity and Relation Extraction with Transformer Pre-training) is also an advanced joint extraction method without knowledge graph context. It attaches entity labels for spans derived from the text, and traversed span pairs for relation judgement (Eberts and Ulges, 2020). • KB-graph is an advanced joint extraction method, which injects medical knowledge into entity and relation extraction and coreference resolution simultaneously (Verlinden et al., 2021). • KECI (Knowledge-Enhanced Collective Inference) is an advanced joint extraction method, which extracts entity and relation collectively with
4http://www.openkg.cn/dataset/medical
knowledge-enhanced collective inference. It utilizes RGCN algorithm for knowledge graph representation (Lai et al., 2021). Compared with the proposed method, KECI merely models the firstorder graph for knowledge fusion. • KECI_nnconv is the variant version of KECI (Lai et al., 2021) in which the NNConv (Gilmer et al., 2017) is utilized instead of RGCN. Different from RGCN, NNConv fuses the initial edge feature for message propagation and further enhances the node representation, which is similar to GEANet. • KECI_gea is one variant version of KECI (Lai et al., 2021) that the RGCN part is replaced by GEANet (Huang et al., 2020) which is similar to NNCONV. • KECI_gn is also one variant version of KECI (Lai et al., 2021) where the node and edge features are simultaneously derived from the graph net (GN) framework (Battaglia et al., 2018).

heading: 4.1.3 Evaluation metrics
text: We evaluate methods by Micro-F1 scores for entity and relation extraction. Also, we use the average Micro-F1 scores of entities and their relations in each medical text as the overall scores for each medical text.

heading: 4.1.4 Implementation details
text: We implement our method with PyTorch and MedBERT-kd Transformer 5 which is based on the structure of BERT and trained on a mount of Chinese clinic texts. The number of parameters of our model is 135,557,734. We conduct hyperparameter tuning by using a Bayesian optimizer for all the methods on real datasets. The scopes of hyper-parameters are: {16, 32} for batch size, {2e5, 3e-5, 4e-5, 5e-5} for lower learning rate, {1e-4, 2e-4, 5e-4} for upper learning rate.
A coarse-to-fine training process is applied during training process. Firstly, we leave out the
5https://huggingface.co/trueto/ medbert-kd-chinese
knowledge graph fusion part and only employ hidden representations of spans from the local context to train a better downstream task-specific BERT structure. After that, global information from knowledge graph representation is added for a precise training.
We evaluate all models with GPUs on the JIUTIAN Platform of China Mobile Research Institute.

heading: 4.2 Evaluation results
text: We evaluate the effectiveness of all methods on three real datasets in Table 1, where we show the Micro-F1 values of both entities and relations. We also show the averaged overall Micro-F1 values. From Table 1, we make the following observations:
(1) KECI outperforms other joint extraction methods which do not involve medical knowledge graph. For example, KECI has 9.9%, 0.2% and 23.1% improvements compared with CASREL on CHIP-2020, CHIP-2022 and DiaKG datasets, respectively. It shows that the medical knowledge graph is essential for providing more contexts in joint entity and relation extraction.
(2) Our model works better than KECI in overall values. Specifically, our model achieves 1.5%, 1.6% and 0.5% improvements compared with the KECI model on CHIP-2020, CHIP-2022 and DiaKG datasets, respectively. It verifies that highorder graph context could provide more information for text modeling. Moreover, before adding relation context, as shown in Table 1, the Micro-F1 score of the entity for KECI is 74.3 % on CHIP2020 dataset, while our method uses relation context and Micro-F1 is increased to 76.5 %. On average, there are 2.2%, 3.3%, and 1.6% improvements on the three datasets respectively after adding relation contexts. It can be seen that the relation context does have a significant effect on improving the entity value.
(3) Results in CHIP-2022 show that our model
has a little poorer performance than that of PFN in overall score, however, we still prevail much in entity result. This is attributed to the fact that the relation types in CHIP-2022 dataset are all logical types, such as casual and entailment relations, which varies considerably from medical knowledge graph.
We further make detailed evaluation on all the entity types which can be seen in table 2 and 3. Generally, our model achieves a higher F1 score among entity types, especially in the less-train-instance targets, e.g., “部位(part) ”, “其他治疗(other therapy) ”, “其他(others) ” in CHIP-2020 dataset and “Amount”, “Method”, “Pathogenesis” in DiaKG dataset, our model performs almost above 4% better than KECI (Lai et al., 2021). When mentioning CHIP-2022 data, we neglect the entity type results owing to its lack of discrimination between entity types.
We also make comparisons between proposed high-order heterogeneous graph modeling method and some typical graph neural networks (GNN). We can conclude that the high-order graph modeling for both entity and relation update is more efficient than a single order heterogeneous graph modeling for entity representation, even with the extra relation update. Details are as follows:
(1) The proposed model exceeds KECI_nnconv, KECI_gea and KECI_gn by 1.2%, 0.9%, 1.6%, respectively, in CHIP-2020 dataset.
(2) The proposed model surpasses KECI_nnconv, KECI_gea and KECI_gn by 1.6%, 2.4%, 1.4%, respectively, in CHIP-2022 dataset.
(3) The proposed model beats KECI_nnconv, KECI_gea and KECI_gn by 0.6%, 0.3%, 0.1%, respectively, in DiaKG dataset.

heading: 4.3 Ablation Study
text: We conduct an ablation study on our model to evaluate the effectiveness of its main modules in Figures 3. Specifically, we compare our model with the following variants w.r.t. the overall score of each medical text, i.e., average Micro-F1 score of entities and relation in each medical text. • MnoKG (Model without Knowledge Graph) ignores knowledge graph for triplets extraction. • MEnt (Model with Entity) merely fuses firstorder graph into medical text for joint extraction. • MRel (Model with Relation) merely fuses second-order graph into medical text for extraction. • MnoGNN (Model without GNN update) fuses initial node representations of high-order graph from TransE into text without GNN to update. • MRAtt (Model with Relation Attention fusion) is one variant version of our model, which uses an attention mechanism in Equation 20 to replace Equation 16 for relation fusion:
fir = Wirfcr(hi) + ∑
rj∈C(sir ) Wijrfr(rj) , (20)
where Wir and Wijr are similar to 9 and 10, respectively.
From the Figures, we make the following observations:
(1) Our model beats MnoKG on all real datasets. In particular, the performance improvements of our model are 8.9%, 6.3% and 5.9% better compared with MnoKG on CHIP-2020, CHIP-2022 and DiaKG datasets, respectively. It shows that using medical knowledge graph is important for providing external contexts.
(2) Our model outperforms MEnt. For example, the overall improvements are 1.5%, 1.6% and 0.5% compared with MEnt on CHIP-2020, CHIP-2022 and DiaKG datasets, respectively. It justifies that the second-order graph provides a positive feedback for the first-order representations and further enhance the encoding of entity mentions in medical text.
(3) Our model surpasses MRel and the rate is less than that of MEnt. It seems the second-order graph plays a major role in dual heterogeneous graph for joint extraction task.
(4) Our model precedes MnoGNN. Specifically, the exceeding values are 1.2%, 2.5% and 0.1% for the datasets. It shows that the GNN update part is vital for a better knowledge graph information fusion.
(5) The MEnt model only added the first-order graph, and the MRel only added the second-order graph. It can be seen from the data that the effect of MRel is better than that of MEnt. Second-order graph can improve entity extraction even more effectively, because it acts directly on the entity span.
(6) Our model performs better than MRAtt. To be specific, the overall improvement of our model is 0.5%/1.5% compared with MRAtt on CHIP2020/CHIP-2022 datasets and the result is comparable to that of MRAtt on DiaKG dataset. It means that the selective gate mechanism can improve the performance for relation fusion.

heading: 5 Conclusions
text: In this paper, we study the problem of the joint entity and relation extraction in the medical text. We propose to construct the high-order heterogeneous graph from the medical knowledge graph, and learn the entity span representations in a knowledgeenhanced which integrates the deep representations from both global and local contexts. The extraction of entities and relations is in a collective way. The experimental results show that our model is more effective than state-of-the-art methods.
Limitations
We inject the medical knowledge graph into local texts for entity span representations enhancement. However, unlike most joint extraction methods, the proposed model is hard to be trained in a parallel way. Therefore, it is time-consuming to obtain a well-trained model. We would like to optimize the architecture of the model in the future.
Moreover, our model is adapted to Chinese medical texts where a token usually means a character (not a word). Hence, there will be errors when aligning the entities from the knowledge graph with mentions from local texts. Word segmentation task will be considered in our future work.

heading: Acknowledgements
text: This work is supported by China Mobile Holistic Artificial Intelligence Major Project Funding (R22105ZS, R22105ZSC01) and National Key R&D Program of China (2021ZD0140408).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: the "Limitations" section which is next to the section 5

heading: 3 A2. Did you discuss any potential risks of your work?
text: the "Limitations" section which is next to the section 5

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: the 1st section (introduction) and the "abstract" section which is ahead of section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? section 4 (experiments)

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4.1.4 (Implementation details)
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4.1.4 (Implementation details)
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? the results are stable without large error bars
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? not used
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We would like to optimize the architecture of the model in the future. Moreover, our model is adapted to Chinese medical texts where a token usually means a character (not a word). Hence, there will be errors when aligning the entities from the knowledge graph with mentions from local texts. Word segmentation task will be considered in our future work