GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs
Authors: Pengcheng Jiang
Cao Xiao
Adam Richard Cross
Jimeng Sun
Venue: ICLR 2024 poster (2024)
Keywords: EHR Prediction
Personalized Knowledge Graph
Graph Neural Network
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Clinical predictive models often rely on patients’ electronic health records (EHR), but integrating medical knowledge to enhance predictions and decision-making is challenging. This is because personalized predictions require personalized knowledge
graphs (KGs), which are difficult to generate from patient EHR data. To address this, we propose GraphCare, an open-world framework that uses external KGs to improve EHR-based predictions. Our method extracts knowledge from large language models (LLMs) and external biomedical KGs to build patient-specific KGs, which are then used to train our proposed Bi-attention AugmenTed
(BAT) graph neural network (GNN) for healthcare predictions. On two public datasets, MIMIC-III and MIMIC-IV, GraphCare surpasses baselines in four vital healthcare prediction tasks: mortality, readmission, length of stay (LOS), and drug recommendation. On MIMIC-III, it boosts AUROC by 17.6% and 6.6% for mortality and readmission, and F1-score by 7.9% and 10.8% for LOS and drug recommendation, respectively. Notably, GraphCare demonstrates a substantial edge in scenarios with limited data availability. Our findings highlight the potential of using external KGs in healthcare prediction tasks and demonstrate the promise of GraphCare in generating personalized KGs for promoting personalized medicine.

Published as a conference paper at ICLR 2024
GRAPHCARE: ENHANCING HEALTHCARE PREDIC- TIONS WITH PERSONALIZED KNOWLEDGE GRAPHS
Pengcheng Jiang' Cao Xiao
Adam Cross
Jimeng Sun
"University of Illinois Urbana-Chane Chaaaa-Chammaagn
1GE HealthCare
*OSF HealthCare
ABSTRACT
Clinical predictive models often rely on patients' electronic health records (EHR), but integrating medical knowledge to enhance predictions and decision-making is challenging. This because personalized predictions require personalized knowledge graphs (KGs), which are difficult to generate from patient EHR data. To address this, we propose GRAPHCARE, a framework that uses external KGs improve EHR-based predictions. Our method extracts knowledge from large language models (LLMS) and external biomedical KGs to build patient-specific KGs, which are then used to train proposed Bi-attention AugmenTed (BAT) graph neural network (GNN) for healthcare predictions. On two public datasets, MIMIC-III and MIMIC-IV, GRAPHCARE surpasses baselines in four vital health- care prediction tasks: mortality, readmission, length of stay (LOS), and drug rec- ommendation. On MIMIC-III it boosts AUROC by 17.6% and 6.6% for mortality and readmission, and F1-scors by 7.9% and 10.8% for LOS and drug recommen- dation, respectively. Notably, GRAPHCARE demonstrates a substantial edge scenarios with limited data. Our findings highlight the potential of using external KGs in healtheare prediction tasks and demonstrate the promise of GRAPHCARE in generating personalized KGs for promoting personalized medicine.
INTRODUCTION
The digitization of healthcare systems has led to the accumulation of vast amounts of electronic health record (EHR) data that encode valuable information about patients, treatments, etc. Machine learning models have been developed on these data and demonstrated huge potential for enhancing patient and resource allocation via predictive tasks, including mortality prediction (Blom al., 2019; Courtright et al., 2019), length-of-stay (LOS) estimation (Cai et al., 2015; Levin et al., 2021), readmission prediction (Ashfaq et al., 2019; Xiao et al., 2018), and drug recommendations (Bhoi et al., 2021; Shang et al., 2019b).
To improve predictive performance and integrate expert knowledge with data insights, clinical knowledge graphs (KGs) were adopted to complement EHR modeling (Chen et al., 2019; Choi et al., 2020; Rotmensch et al., 2017). These KGs represent medical concepts (e.g., diagnoses, procedures, drugs) and their relationships, enabling effective learning of patterns and dependencies. However, existing approaches mainly focus on simple hierarchical relations (Choi et al., 2017; 2018; 2020) rather than leveraging comprehensive relationships among biomedical entities despite incorporating valuable contextual information from established biomedical knowledge bases (e.g., UMLS (Bo- denreider, 2004)) could enhance predictions. Moreover, large language models (LLMs) such as GPT (Brown et al., 2020; Chowdhery et al., 2022; Luo et al., 2022; OpenAI, 2023) pre-trained on web-scale biomedical literature could serve alternative resources for extracting clinical knowl- edge given their remarkable reasoning abilities on open-world data. There is a substantial body of existing research demonstrating their potential use as knowledge bases (Lv et al., 2022; Petroni et al., 2019; AlKhamissi et al., 2022).
To fill the gap in personalized medical KGs, we propose leverage the exceptional reasoning abili- ties of LLMs to extract and integrate personalized KG from open-wone open-world data. Our proposed method GRAPHCARE (Personalized Graph-base based HealthCare Prediction) is a framework designed gen- erate patient-specific KGs by effectively harnessing the wealth of clinical knowledge. As shown
@ @illinois.edu

---

Published as a conference paper at ICLR 2024
Medical Concepts
conditions
C C2, .... Cy
procedures
Pi. P2.......,.. Pr
drugs
d1, d2 ...,
generation
peneration
G
Concept- specific KGs
Node & Edge Clustering
Node Edge Clustering
Mode & Edge Clustering
Cm
G Pt
G
pa
G P-
G'
G
Patients
Patient Patient-Graphh.ooo Composition
Healthcare Predictions Pradicttan b-B
Bi-attention Augnented
Patient Graph
Corneiluiion
Arttvvatton
Length Stay
oa
06
aEt.
1E4COUT
0
88
Rataand guai unteliigg
Predlctiion
Sequential
dngpost
condition events
aedication events
procedore events
0
HLP
Figure 1: Overview of GRAPHCARE. Above: Given a patient record consisting of conditions, procedures and medications, we generate a concept-specific KG for each medical concept, by knowledge probing from LLM and subgraph sampling from cxisting KG; and we perform node and edge clustering among all graphs (53.1). Below: each patient, we compose a patient-specific graph by combining the concept-specific KGs associated with them and make the graph temporal with sequential data across patient's visits ($3.2). To utilize the patient graph for predictions, we employ bi-altention augmented graph neural network (GNN) model, which highlights essential visits and nodes with attention weights ($3.3). With three types of patient representations (patient-node, patient-grape and joint embeddings), GRAPHCARE capable of handling a variety of healtheare predictions ($3.4).
in Figure 1, our patient KG generation module first takes medical concepts as input and generates concept-specific KGs by prompting LLMSs retrieving subgraphs from existing graphs. It then clus- ters nodes and edges to create a more aggregated KG for each medical concept. Next, constructs personalized KG for each patient by merging their associated concept-specific KGs and incorpo- rating temporal information from their sequential visit data. These patient-specific graphs are then fed into our Bi-attention AugmenTedd (BAT) graph neural network (GNN) for diverse downstream prediction tasks.
We evaluated the effectiveness of GRAPHCARE using widely-used EHR datasets, MIMIC-II (Johnson et al., 2016) and MIMIC-IV (Johnson et al., 2020). Through extensive experimentation, we found that GRAPHCARE outperforms several baselines, while BAT outperforms state-of-the-at GNN models (Veličković et al., 2017: Hu et al., 2019; Rampášek et al., 2022) on four common healthcare prediction tasks: mortality prediction, readmission prediction, LOS prediction, and drug recommendation. Our experimental results demonstrate that GRAPHCARE, equipped with the BAT, achieves average AUROC improvements of 17.6%, 6.6%, 4.1%. 2.1% and 7.9%, 3.8%, 3.5%, 1.8% over all baselines MIMIC-III and MIMIC-IV, respectively. Furthermore, our approach requires significantly fewer patient records to achieve comparable results, providing compelling evidence for the benefits of integrating open- world knowledge into healthcare predictions.
2 RELATED WORKS
Clinical Predictive Models. EHR data has become increasingly recognized as a valuable resource in the medical domain, with numerous predictive tasks utilizing this data (Ashfaq et al., 2019; Bhoi et al., 2021; Blom et al., 2019; Cai et al., 2015). A multitude of deep learning models have been designed to cater to this specific type of data, leveraging its rich, structured nature to achieve en- hanced performance (Shickel et al., 2017: Miotto et al., 2016; Choi et al., 20;6c;;;;; Shang et al., 2019b; Yang et al., 202 Choi et al., 2020; Zhang et al., 2020; Ma et al., 2020b;a; Gao et al., 2020; Yang al., 2023b). Among these models, some employ graph structure to improve prediction accuracy, effectively capturing underlying relationships among medical entities (Choi et al., 2020; Su et al., 2020; Zhu & Razavian, 2021; Li et al., 2020; Xie al., 2019; Lu et al., 2021 a,; Yang al.,
2

---

Published as a conference paper at ICLR 2024
2023b; Shang al., 2019b). However, limitation of these existing works is that they do not link the local graph to an external knowledge base, which contains a large amount of valuable relational information (Lau-Min et al., 2021: Pan & Cimino, 2014). We propose create customized knowl- edge graph for each medical concept in an open-world setting by probing relational knowledge from either L.LMs or KGs, enhancing its predictive capabilities for healthcare.
Personalized Knowledge Graphs. Personalized KGs have emerged as promising tools for improv- ing healthcare prediction (Ping et al., 2017: Gyrard et al., 2018; Shirai et al., 2021; Rastogi & Zaki, 2020; Li et al., 2022). Previous approaches such as GRAM (Choi et al., 2017) and its successors (Ma et al., 2018; Shang et al., 2019a; Yin al.. 2019; Panigutti al., 2020; Lu et al., 2021b) incorporated hierarchical graphs to improve predictions of deep learning- based models; however, they primarily focus on simple parent-child relationships, overlooking the rich complexities found in large knowledge bases. MedMI. (Gao et al., 2022) employs graph data for COVID-19 related prediction. However, the KG in this work has a limited scope and relies heavily on curated features. To bridge these gaps, introduce two methods for creating detailed, personalized KGs using open sources. The first solution is prompting (Liu et al., 2023) LLMs to generate KGs tailored to medical concepts. This approach is inspired by previous research (Yao et al., 2019; Wang al., 2020a; Chen et al., 2022; Lovelace & Rose, 2022: Chen et al., 2023; Jiang et al., 2023), showing that pre-trained LMs can function comprehensive knowledge bases. The second method involves subgraph sam- pling from established KGs (Bodenreider, 2004), enhancing the diversity of the knowledge base.
Attention-augmenter GNNs. Attention mechanisms (Bahdanau et al., 2014) have been widely utilized in GNNs to capture the most relevant information from the graph structure for various tasks (Veličković et al., 2017; Lee al.. 2018: Zhang et al., 2018; Wang et al., 2020b; Zhang et al., 2021a; Knyazev et al., 2019). The incorporation of attention mechanisms in GNNs allows for enhanced graph representation learning, which is particularly useful in the context of EHR data analysis (Choi et al., 2020; Lu et al., 2021b). In GRAPHCARE, we introduce new GNN BAT leveraging both visit- level and node-level attention, edge weights, and attention initialization for EHR-based predictions with personalized KGs.
PERSONALIZED GRAPH-BASED HEALTHCARE PREDICTION
In this section, we present GRAPHCARF, a comprehensive framework designed to generate person- alized KGs and utilize them for healthcare predictions. It operates through three general steps:
Step I: Generate concept-specific KGs for every medical concept using LLM prompts and by sub- sampling from existing KGs. Perform clustering on nodes and edges across these KGs.
Step 2: For each patient, merge relevant concept-specific KGs to form a personalized KG.
Step 3: Employ the novel Bi-attention Augmented (BAT) Graph Neural Network (GNN) to predict based on the personalized KGs.
3.1 STEP CONCEPT-SPECIFG KNOWLEDGG GRAPH GENERATION.
{c. p, d]. where = (C1, C2, ..., C.)), p (p1, Pe.......... and d d (d1, d2, ..., dia)) correspond to sets of medical concepts for conditions, procedures, and drugs, with sizes of |c). Ip). and |d]. respectively. The goal of this step is to generate a KG G = (V", g*) for each medical concept e, where y represents nodes, and E" denotes edges in the graph.
Denote medical concept
Our approach comprises two strategies: (1) LLM-based KG extraction via prompting: Utiliz- ing a template with instruction, example, and prompt. For example, with an instruction "Given prompt, extrapolate as many relationships as possible of it and provide a list of updates". an example "prompt: systemic lupus erythematosus. updates: [systemic lupus erythematosus is, treated with, steroids]..." and a prompt "prompt: tuberculosis. updates:", the LLM would respond with list of KG triples such as "[tuberculosis, may be treated with, antibiotics], [tuberculosis, affects, lungs]..." where each triple contains head entity, a relation, and a tail entity. Our curated prompts are de- tailed in Appendix D. After running x times, we aggregate' and parse the outputs to form a KG for each medical concept, Ge GLLM))) (Vi (PLMMO)--LLMCC
(2) Subgraph sampling from existing
'To address ethical concerns with L.M use, collaborate with medical professionals to evaluate the ex- tracted KG triples, which minimizes the risk of including any inaccurate or potentially misleading information.
3

---

Published as a conference paper at ICLR 2024
Leveraging pre-existing biomedical KGs (Belleau al., 2008; Bodenreider, 2004; Donnelly et al., 2006), we extract specific graphs for a concept via subgraph sampling. This involves choosing relevant nodes and edges from the primary KG. For this method, first pinpoint the entity in the biomedical KG corresponding to the concept e. We then sample a x-hop subgraph originated from the entity, resulting in G aub(()) D.2. Consequently, for each medical concept, the KG is represented as G G(.M())) U G sub(x))
KGs:
Subbky & s)b(s)) We detail the sampling process in Appendix
Node and Edge Clustering. Next, we perform clustering of nodes and edges based on their sim- ilarity, to refine the concept-specifie KGs. The similarity is computed using the cosine similarity between their word embeddings. We apply the agglomerative clustering algorithm (Müllner, 2011) on the cosine similarity with a distance threshold ó, to group similar nodes and edges in the global graph G = (G G G G lel ())))) of all concepts. After the clustering process, we obtain Cy : V + V and Cg : E --5 E' which map the nodes V and edges & in the original graph G to new nodes V' and edges E", respectively. With these two mappings, we obtain a new global graph G' = (V', E.), and we create a new graph G'= = "oe, C G' for each concept. The node embedding HD E RIv'Xx and the edge embedding H E B|E"|x are initialized by the averaged word embedding in each cluster, where w denotes the dimension of the word embedding.
3.2 STEP 2: PERSONALIZED KNOWLEDGE GRAPH COMPOSITION
For each patient, we compose their personalized KG by merging the clustered KGs of their medical concepts. We create a patient node (P) and connect it to its direct EHR nodes in the graph. The personalized KG for patient can be represented as Gpat U (E'e1 g'ee.".... E'.,,,, with (ei, (2.,;e... ew)
(Vpat: Epat), where
Ypac P ( , being the medical concepts directly associated with the patient, w being the number of concepts, and e being the edge connecting P and {El, e2,...,
V'ea V'e,,...VV V'e.) and Epat
Further, as a patient is represented as a sequence of J visits (Choi et al., 2016a), the visit-subgraphs for patient i can be represented as (G Gi...... G.,,} = E111) (V E,,, .... (Vi,J, E,,,)} for visits {I1, T2, T2, ..., xJ} where Vi S (iatte) and ELg S Epan()) for 1 S j S J. We introduce Emter for the interconnected- ness across these visit-subgraphs, defined as: Eister = {(ras, < wir () E Vagi VLJ', ' Vigri + j"; and (vuj, + w.) E'). This set includes edges (Vi.j. ++ vg.j(,e) ) that con- nect nodes Vi,jA and Va from different vusit-subgraphs Gt, and Gi,j' respectively, provided that there exists edge (Vigii Va xe) in the global graph G'. The final representation of the patient's personalized KG, Gpa(t). integrating both the visit-specific data and the broader inter-visit
connections, is given as: Gpa(t)) (P (PUU,= Viy,e (U/=1 EL.)) U Uinter)
3.3 STEP 3: BI-ATTENTION A AUGMENTER GRAPH NEURAL NETWORK
Given that each patient's 's data encompasses multiple risit-subgraphs. becomes imperative devise specialized model capable of managing this temporal graph data. Graph Neural Networks (GNNs), known for their proficiency in this domain, can be generalized as:
(1+1) (w()) AGGREGATE(() (b IK' EN(K)) + bb)). h
(1)
where h (1+1) represents the updated node representation of node k at the (1 + 1)-th layer of the GNN. The function AGGREGATE aggregates the node representations of neighbors A(k)) of node k at the I-th layer. ( and (()) are the learnable weight matrix and bias vector at the I-th layer, respectively. denotes activation function. Nonetheless, the conventional GNN approach overlooks the temporal characteristics of our patient-specific graphs and misses the intricacies of personalized healthcare. To address this, we propose a Bi-attention Augmented (BAT) GNN that better accommodates temporal graph data and offers more nuanced predictive healthcare insights.
Our model. In GRAPHCARE, information from the personalized KG. We first reduce the size of node and edge embedding from the word embedding the hidden embedding to improve model's efficiency. The dimension-reduuced hidden embeddings are computed as follows:
incorporate attention mechanisms to effectively capture relevant
hijik = W. W.h (15.)) + by (aga)))()) = W.h (.iheeee + by
(2)
4

---

Published as a conference paper at ICLR 2024
where W., W, E R"x,, b b, E R' are learnable vectors, h (..,)) h (k. R input embedding, hij.k, Rij subgraph of patient, and the hidden embedding of the edge between the nodes U.j.k and Ui respectively. q is the size of the hidden embedding.
E R2 hidden embedding of the k-th node in j-th visit-
Subsequently, we compute two sets of attention weights: one set corresponds to the subgraph as- sociated with each visit, and the other pertains to the nodes within each subgraph. The node-level attention weight for the k-th node in the j-th visit-subgraph of patient i, denoted as a4,j,k, and the visit-level attention weight for the j-th visit of patient i, denoted as Br.j. are shown as follows:
ai aij.M M- Softmax Wigiij + bo):
,, B,, = X Tanh(w w G; + by), where A A=
(3)
where S E RM is multi-hot vector representation, of visit-subgraph Gi.. indicating the nodes appeared for the j-th visit of patient i where M = [ is the number of nodes in the global graph G. G, E N represents the multi-hot matrix of patient i's graph G3 where N is the maximum visits across all patients. Wa E IR MMM, Wg E 18 ad ba E R² and by E IN are learnable parameters. À E R is the decay coefficient vector, is the number of visits of patient i, A E A where \3 exp (->(J - j)) when j S J and otherwise, is coefficient for the visit j, with decay rate initializing higher weights for more recent visits.
Attention initialization. To further incorporate prior knowledge from LLMs and help the model converge, we initialize Wa for the node-level attention based on the cosine similarity between the node embedding and the word embedding Wi of a specific term for the prediction task-feature pair We provide more details on this in Appendix C). Formally, we first calculate the weights for the nodes in the global graph G' by Wm (h.. ( l|w where h H is the input embedding of the m-th node in G', and w.m the weight computed. We normalize the weights s.t. 0 S wm I 1, 1 I m S M. We initialize W. diag(w), (w.,..., as a diagonal matrix.
(e.g., "terminal condition" for
Next, we update the node embedding by aggregating the neighboring nodes across all visit- subgraphs incorporating the attention weights for visits and nodes computed in Eq (3) and the weights for edges. Based on Eq (1), we design our convolutional layer BAT as follows:
(+11 jifk =
W(
Σ
L der R(kkk)) h ((JA))))))))
+ b
Node aggregation term
Edge aggrogation torm
(4)
where is the ReLU function, W(O E R b(D) Ra are learnable parameters, WR glE'l (f)
(1) is the scalar weight for the edge the edge weight vector the layer I, and w W)( E w
embedding h (te.,.hee In Eq (4), the node aggregation term captures the contribution of the attention- aetention-weighted nodes, while the edge aggregation term represents the influence of the edges connecting the nodes. This convolutional layer integrates both node and edge features, allowing the model to learn rich representation of the patient's 's EHR data. After several layers of convolution, we obtain the node embeddings h (2)
k.jkk of the final layer (L), which are used for the predictions:
K
K3
h
Gpar
MEAN
ΣΣ
h = -EEAN
ΣΣ ===1
k.j.ke
1=
graph MLP(h, Gpat
pode MLP(h?),
joint MLP(h,
Grar + h?),
(5)
where J is the number of visits of patient i, K, is the number of nodes in visit j. h h. denotes the patient graph embedding obtained by averaging the embeddings of all nodes across visit-subgraphs and the various nodes within each subgraph for patient i. h." represents the patient node embedding computed by averaging node embeddings of the direct medical concept linked to the patient node. i.j.k E (0, 1} is binary label indicating whether node Ui.j.k corresponds to a direct medical concept for patient i. Finally, we apply multilayer perception (MLP) to either h Gpas hf, the concatenated embedding (h patt + h,") to obtaín logits z graph z,nde or z joint respectively. We discuss more details of the patient representation learning in Appendix E.
A
5

---

Published as a conference paper at ICLR 2024
Table I: Statistics of pre-processed EHR datasets. "#" "the number of", "/patient": "per patient".
opatients
#visits
#visits/patient
#conditions/patient
Mprocedures/paaient
#drruss/ptient
MIMIC-III
35,707
44,399
1.24 1.88
12.89
4.54
33.71
MIMIC-IV
123,488
232,263
21.74
4.70
43.89
3.4 TRAINING AND PREDICTION
The model can be adapted for a variety of healthcare prediction tasks. Consider a set of samples {(x1), (x1, ..... (Ii, 22,,... ] for each patient with visits, where each tuple represents sample consisting of sequence of consecutive visits.
Mortality (MT.) prediction predicts the mortality label of the subsequent visit for each sample, with the last sample dropped. Formally, f (I1, :2...., II--) - y(2z] where y|æz|| E {0, is binary label indicating the patient's 's survival status recorded in visit z4.-
Readmission (RA.) prediction predicts if the patient will be readmitted into hospital within o - v(T(x)) 7xx y E (0,1) where T(2e) denotes the encounter time of visit Ft. v/r(2))) (()--1)] equals 1 if T(xe) - T(x--1) S a, and 0 otherwise.
days. Formally, f : (#1, ...,...,
In our study, we set a = 15 days.
Length-Or-Stay (LOS) prediction (Harutyunyan et al., 2019) predicts the length of ICU stays for each visit. Formally, ʃ (21, #2,...., : .et) + y|zre] where y [1 E RIXC is one-hot vector indicating its class among C' classes. We set 10 classes [0, 10.1.... 7, 9], which signify the stays of length < day (0), within one week (1...., 7), one to two weeks (8), and 2 two weeks (9).
Drug recommendation predicts medication labels for each visit. Formally, f (æ1, 22,.. ........ It) y|ar)] where y[ara] E 1 dl is multi-hot vector where |d| denotes the number of all drug types.
We use binary cross-entropy (BCE) loss with sigmoid function to train binary (MT. and RAA.) and multi-label classification (Drug.) classification tasks, while we use corss-entropy (CE) loss with softmax function to train multi-class (LOS) classification tasks.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Data. For the EHR data, we use the publicly available MIMIC-III (Johnson et al., 2016) and MIMIC-IV (Johnson al., 2020) datasets. Table presents statisties of the processed datasets. To build concept-specific KG ($3.1), we utilize GPT-4 (OpenAI, 2023) the LLM for KG generation, and utilize UMLS-KG (Bodenreider, 2004) as an existing biomedical KG for subgraph sampling. which has 300K entities and 1M relations. X 3 and K 1 are set parameters. We employ the GPT-3 embedding model to retrieve the word embeddings of the entities and relations.
Baselines. Our baselines include GRU (Chung et al., 2014), Transformer (Vaswani et al., 2017), RETAIN (Choi al., 2016c), GRAM (Choi et al., 2017), Deepr (Nguyen et al., 2016), StageNet (Gao et al., 2020), AdaCare (Ma et al., 2020a), GRASP (Zhang et al., 2021b), SafeDrug (Yang al., 2021b), MICRON (Yang et al., 2021a), GAMENet (Shang al., 2019b), and MoleRec (Yang et al., 2023b). AdaCare and GRASP are evaluated only on binary prediction tasks given their com- putational demands. For drug recommendation, we also consider task-specific models SafeDrug, MICRON, GAMENet, and MoleRec. Our GRAPHCARE model's performance is examined under five GNNs and graph transformers: GAT (Veličković et al., 2017). GINE (Hu al., 2019), EGT (Hussain et al., 2022), GPS (Rampášek et al., 2022) and our BAT. We do not compare to models such as GCT (Choi et al., 2020) and CGL (Lu et al., 202 la) as they incorporate lab results and clin- ical notes, which are not used in this study. Implementation details are discussed in Appendix C.
Evaluation Metrics. We consider the following metrics: (a) Accuracy the proportion of correctly predicted instances out of the total instances; (b) F1 the harmonic mean of precision and recall; (c) Jaccard score the ratio of the intersection to the union of predicted and true labels; (d) AUPRC the area under the precision-recal curve, emphasizing the trade-off between precision and recall; (e) AUROC the area under the receiver operating characteristic curve, capturing the trade-off between the true positive and the false positive rates. (f) Cohen's Kappa measures inter-rater agreement for categorical items, adjusting for the expected level of agreement by chance in multi- class classification.

---

Published as a conference paper at ICLR 2024
Table 2: Performance comparison of four prediction tasks on age performance (%) and the standard deviation (in bracket) of each model over 100 runs for MIMIC-II and 25 runs for MIMIC-IV. The best results are highlighted for both datasets.
We report the
Task I: Mortality Prediction
Task 2: Readmission Prediction
Model
MIMIC-III
MIMIC-IV
MIMIC-III
MIMIC-IV
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
GRU
11 11.8
61.3
4.2g0.1)
69.0ro.s
68.2(u.4)
65.4
66.1go1)
(6.2(o.3)
Transformer
(0.4
57 27.2
3.4(0.4)
G5.1/1.2)
67.3
63.9(1.)
65. 65.7
65.3
RETAIN
9.
59.4
3.8(o.0)
64.8q 6t
65.1(1.0)
64.1(
66 ((0.3)
66.3
GRAM
11.40.77
60.4g.m)
1.4u.a)
66.7
67.2
61.3(.4)
66.1(0.2)
66.3
Deepr
13.2
60.8(a.4)
4.2r0.9)
68.9ro.
68.8(0.8)
66.5(a..)
65 65.6(a.1)
65.4
AdaCure
11.1
58 68.4p.44
4.6
69.3
68.6
65.7(u3)
65 65.9(u.)
66 66.1
GRASP
(.9
(9.2(1.4)
4.7
68.44..
69.2(g.4
66.3(p.0)
66.3(a.3)
66.1
StageNet
12.4
61.5gu.7)
4.2g0.3)
69.6ro.8)
69.3
66.7(u.4)
66.1
66.2
GRAPHCARE
w/ GAT
14.3,0.8)
67.8
5.1
71.8:10;
71.5
(8.7(a.))
67.4(0.4)
67.3
w/ GINE
14.4(0.4)
67.3
5.7
72.0:1.3)
71.3
(8.0
68.3 68.3
67.5
w/ EGT
15.5
69.1
6.2
71.3
72.2(u.5)
68 S(u.4)
68.9().2)
67. 07.66..)
w/ GPS
15.3
68. 68.8
6.7
72.7
71.9(a.6)
68.5
69.1(o.4)
67. 67.9(o.4)
w/ BAT
(6.7(0.5)
70.3
(.7
73.1.0.))
73.4(0.4)
69.7
69. 6..)
68. 68.5
Task 3: Length of Stay Prediction
Model
MIMIC-II
MIMIC-IV
AUROC
Kappa
Accuracy
Fl-score
AUROC
Kappa
Accuracy
Fi-score
GRU
78.3
(6.2
40.3
34.9ro.s
78.7
26.0(o.1)
35.2
31.6(o2)
Transformer
78.3,0.2)
25.4
40.1.o.3)
34.8
75.3
25.3(0.4)
34.4...2)
31.40.3)
RETAIN
78.2
26.1
40.6
34.9/o.4)
78.9
26.3
35.7
32.0
GRAM
(8.2
26.3
10.4/0.4)
31.5%o
78.8
(6.4()
35 (5.....))
31.9(o.)
Deepr
77.9(0.1)
25.3/0.4)
40.
35.0/o.4)
79.5(a.a)
26.4(n.2)
35.8/o.a)
32.3
StageNet
78.3
24.8(o.a)
39.9(o.2)
31.4/o44
79.2(u.3)
26.0(ua)
35. 35.0(o.a)
31.3
GRAPHCCRE
w/ GAT
79.4(0.3)
28.2(a2)
41.9
36.1
80.3,0.3
28. 28.4(0.4)
36.2(0.1)
33.3
w/ GINE
79.2gu.3)
28.3 28.3
41.
36.0ro.4:
79.9
27 27.30
36 36.3
32.8p
80.3(o.3)
28.8(a.2)
42.8
36.3
80.5go.3)
28-7(0.5)
(6.7(0.2)
38.5
w/ GPS
80.9
28. 28.8(0.4)
43.0(o.s)
36.8:0.4;
80.7(a.a)
28. 2(..4)
36.7,0.3)
33.9/o.3)
w/ BAT
81.4.0.3)
29.5.0.4)
43.2(0.4)
37.5.0.2)
81.7
29.8
37 (7.3 34.2
Task 4: Drug Recommendation
Model
MIMIC-III
MIMIC-IV
AUPRC
AUROC
Fi-score
Jaccard
AUPRC
AUROC
Fi-score
Jaccard
GRU
77.0ga.1)
94.4
62.3
47.8
74.1
(4.2(0.1)
(0.2(0.))
44.0
Transformer
76.1...)
94. 94.2
62.1
f7.1
71.3
93.4ga.)
55 55.9(o.8)
40.4(.1)
RETAIN
77.1
94.4
63.7
48.8
74.2(0.1)
94.3
60.3
45.0
GRAM
(6.7
94.2(a.1)
62.9/o.)
47.9ro.3)
74.3g0.3
94.2(0.1)
60.1
45.3...5
Deepr
74.3
93.7,0.0)
60.3
44.7:0.s;
73.7
94.2
59.1
43.8(0.4)
Stage/Net
74.4
93-00a..)
61.4(o.5)
45.8
74.4.0.1)
94.2(0.0
60.2(0.3)
45.4(0.)
SafeDrug
68.1
91.0/o.1)
46.7,0.4)
31.7
66.4(a.6)
91.8(p.2)
56.2(0.4)
44.3
MICRON
77.4ç0.0)
94.6(o.1)
63.2(0.4)
48.3:04;
74.4(0.1)
94.3(..))
59.3
44.1
GAMENet
76.4g.0)
94.2(0.1)
62.L(o.4)
47.2
74.2
(41.(
60.4go.4)
45.3
MoleRec
69.8
92.0/o.13
58.1
43.1
08.6ro.1)
92.1
56.3
40.6
GRAPHCARE
w/ GAT
78.5
94.8(a.)
64.4..5)
49.2
71.7
91.4(a.))
60. 60.4go.)
45.7p..)
GINE
78.2
94.7(u.)
63.6
47.9ro
74.8
94.6(aa))
60 6(0.4)
46.1
w/ EGT
79.6,0.2)
95.3
66.4..22)
49.6:0.4;
75.4(0.4)
95.0(a.))
61.6(a.5)
47.3
w/ GPS
79.9(a.3)
95.5.a.1)
66.2
49.8,04)
75.9ka.5)
94.9(a.))
62.1 (2.1
46.8p.4)
w/ BAT
80.2
95.5:0.))
66. 66.8
49.7
77.1
95.4:
63 63.9/0.a)
48 1
4.2 EXPERIMENTAL RESULTS
As demonstrated in Table 2, GRAPHCARE consistently outperforms existing baselines across all prediction tasks for both MIMIC-II and MIMIC-IV datasets. For example, when combined with BAT, GRAPHCARE exceeds StageNet's best result by +1 +4.33 in AUROC for mortality prediction on MIMIC-III Within our GRAPHCARE framework, our proposed BAT GNN consistently performs the best. underlining the effectiveness of the bi-attention mechanism. In the following, we analyze the effects of incorporating the personalized KG and our proposed BAT in detail.
4.2.1 EFFECT OF PERSONALIZED KNOWLEDGE GRAPH
Effect of EHR Data Size. To examine the impact of training data volume on model performance, we conduct a comprehensive experiment where the size of the training data fluctuates between 0. 1.1 and 100% of the original training set, while the validation/testing data remain constant.
7

---

Published as a conference paper at ICLR 2024
LOS Prediction
Drug Recommendaaioon
Performance metrics are averaged over 10 runs, each initiated with a different random seed. The results, depicted in Figure 2, suggest that GRAPHCARE ex- hibits a considerable edge over other models when confronted with scarce training data. For instance, GRAPII- CARE, despite being trained on mere 0.1% of the training data (36 patient samples), accomplished an LOS predic- tion accuracy comparable to the best baseline StageNet that trained on 2.0% of the training data (about 720 patient samples). A similar pattern appears in drug recommendation tasks. Notably, both GAMENet and GRAM also show a certain level of resilience against data limitations, likely due to their use of internal EHR graphs or external ontologies.
Accuracy
AUPRIC
6NU
Tansforer
GAAM
Raka
RETAN
GAAM
Deep
Craph
Training 2e9eeee
222222035 20 woo.oe
Training Size
Performance by EHR training data sizes. Values the x-axis indicate % of the entire training data. The dotted lines separate three ranges: 10.1, 11, [1, 10] and [10, 100] (%).
Figure 2:
Effect of Knowledge Graph Size. Figure illumi- nates how varying sizes of KGs influence the effi- cacy of GR GRAPHCAR.. We test GPT-KG (generated by GPT-4), UMLS-KG (sampled from UMLS), and GPT-UMLS-KG (a combination). Key obser- vations include: (I) Across all KGs, as the size ratio of the KG increases, there is correspond- ing uptick in GRAPHCARE's performance. (2) The amalgamated GPT-UMLS-KG consistently outper- forms the other two KGs. This underscores the premise that richer knowledge bases enable more precise clinical predictions. Moreover, it demon- strates GPT-KG and UMLS-KG could enrich each other with unseen knowledge. (3) The degree of KG contribution varies depending the task at hand. Specifically, GPT-KG demonstrates a stronger influence over mortality and LOS predic- tions compared to UMLS-KG. Conversely, UMLS- KG edges out in readmission prediction, while both KGs showcase comparable capabilities in (4) Notably, lower KG ra- tios (from 0.1 to 0.5) are associated with larger standard deviations, which is attributed to the re- duced likelihood of vital knowledge being encom- passed within the sparsely sampled KGs.
Notta AUROC
Readiiizionn AJROC
6.1
OPEAN
6.1
OPTUNILNN
67
6.7
OPEAT
OPEAT 07TARL54
Figure 3: Performance by different KG sizes.
We test on three distinct KGs: GPT-KG, UMLS- KG, and GPT-UMLS-KG. For cach, we sample sub-KGs using varying ratios: [0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0) while ensuring the nodes correspond- ing to EHR medical concepts remain consistent across samples. The distributions are based on 30 runs on the MIMIC-III with different random seeds.
drug recommendations.
4.2.2 EFFECT OF BI-ATTENTION AUGMENTED GRAPH NEURAL NETWORK
Table 3 provides an in-depth ablation study on the proposed GNN BAT, highlighting the profound influence of distinct components on the model's effectiveness.
The data reveals that excluding node-level attention (a) results in a general drop in performance across tasks for both datasets. This downturn particularly pronounced for the drug recommen- dation task. Regarding visit-level attention (8), the effects of its absence are more discemible in the MIMIC-IV dataset. This is likely attributed to MIMIC-IV's larger average number of visits per patient, as outlined in Table 1. Given this disparity, the ability to discern between distinct visits becomes pivotal across all tasks. Moreover, when considering tasks, it's evident that the RA. task is particularly vulnerable to adjustments in visit-level attention (3) and edge weight (WR). This under- lines the significance of capturing visit-level nuances and inter-entity relationships within the EHR to ensure precise RA. outcome predictions. Regarding attention initialization (Armlnit), it emerges as a crucial factor in priming the model to be more receptive to relevant clinical insights from the get-go. Omitting this initialization shows noticeable decrement in performance, particularly for
8

---

Published as a conference paper at ICLR 2024
Table 3: Variant analysis of BAT We measure AUROC for MT. and RA. prediction, and Fl score for the tasks LOS prediction and drug recommendation. a. (. wR. and Attnni are node-level, visit-level attention, edge weight, and attention initialization, respectively. We report the average performance of 10 runs for each case.
MIMIC-III
MIMIC-IV
Case
Variants
MT.
RA.
LOS
Drug.
MT.
RA.
LOS
Drug.
#0
w/ all
70.3
69.7
37.5
66.8
73.1
68.5
31.2
63.9
#1
w/o a
68.7
68.5
36.71as
64.6
72.2
67.8.a.7
33.1
61.6,23
#2
w/o B
69.9
68.7,1.0
37.2/0.3
66.5ya.
72.1
67.0.1.
33.510.7
63.2
#3
w/o
69.8
68.411.3
36.81a.7
66.3
72.9/o.2
67.9.0.
33.7,0.6
63.
#4
w/o ArmInis
69.5
69.2.p.s
37.2,0.3
65.5g1.3
72.51o.6
68.1.004
34.1
62.4g1.5
#5
w/o #(12233,4)
67 67.4
68.141.6
36.0p1.5
64.0
71.711.4
67.5.1.
32.9
60.5,3.4
drug recommendations. This suggests that by guiding initial attention towards potentially influential nodes in the personalized KG, the model can more adeptly assimilate significant patterss and make informed predictions.
4.3
INTERPRETABILITY OF GRAPHCARE.
d
PATIEN
PATIENT
deally carter
No Pretreate ig
Figure 4: Example showing a patient's personalized KG with importance scores (Appendix F) visualized. For better presentation, we hide the nodes of drugs. The red node represents the patient node. Nodes with higher scores are Warkel and larger. Edges with higher scores darker and thicker. Subgraph (a) shows central area overview of this personalized KG, and other subgraphs show more details with a focused node highlighted.
Figure showcases an example of a personalized KG for mortality prediction tied to a specific pa- tient (predicted mortality 1), who was accurately predicted only by our GRAPICARE method, while other baselines incorrectly estimated the outcome. Figure 4a, important nodes and edges con- tributing to mortality prediction, such "deadly cancer", are emphasized with higher importance scores. This demonstrates the effectiveness of our BAT model in identifying relevant nodes and edges. Additionally, Figure 4b shows the direct EHR nodes connected the patient node, enhanc- ing interpretability of predictions using patient node emhedding. Figure 4c and 4d show KG triples linked the direct EHR nodes "bronchiectasis" and "pneumonia". These nodes are connected to important nodes like "mortality", "respiratory failure", "lung cancer", and "shortness of breath", indicating their higher weights. In Figure 4e, the "lung cancer" node serves as a common connector for "bronchiectasis" and "pneumonia", It is linked to both "mortality" and "deadly cancer", high- lighting its significance. Removing this node had a noticeable impact on the model's performance, indicating its pivotal role in accurate predictions. This emphasizes the value of comprehensive health data and considering all potential health factors, no matter how indirectly connected they may seem.
5 CONCLUSION
We presented GRAPIICAEE, a framework that builds personalized knowledge graphs for enhanced healthcare predictions. Empirical studies show its dominance over baselines in various tasks on two datasets. With its robustness to limited data and scalability with KG size, GRAPHCARE promises significant potential in healthcare. We discuss ethics, limitations, and risks in Appendix A. Our code is available at https:/ / thub. com/ httmm/pat-ia http:pattaa//aaaaaaeaaaeaee

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This work was supported by NSF award SCH-2205289, SCH-2014438, and IIS-2034479.
REFERENCES
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. A review on language models knowledge bases, 2022.
Awais Ashfaq, Anita Sant' Anna, Markus Lingman, and Stawomir Nowaczyk. Readmission pre- diction using deep learning electronic health records. Journal of biomedical informatics, 97: 103256, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint urXiv:1409.000 2014.
Mathieu Bastian, Sebastien Heymann, and Mathieu Jacomy. Gephi: an open source software for exploring and manipulating networks. Proceedings of the international AAAI conference on web and social media, volume 3, pp. 361-362, 2009.
François Belleau, Marc- Alexandee Nolin, Nicole Tourigny, Philippe Rigault, and Jean Morissette. Bio2rdf: towards mashup to build bioinformaties knowledge systems. Journal of biomedical informatics, 7165):706-716, 2008
Suman Bhoi, Mong Li Lee, Wynne Hsu, Hao Sen Andrew Fang, and Ngiap Chuan Tan. Personaliz- ing medication recommendation with a graph-basee based approach. ACM Transactions on Information Systems (TOIS), 40(3):1-23, 2021.
Mathias Carl Blom, Awais Ashfaq, Anita Sant' Anna, Philip D Anderson, and Markus Ling- man. Training machine learning models to predict 30-day mortality in patients discharged from the emergency department: retrospective, population- based registry study. BMJ open, 9(8): e028015, 2019.
Olivier Bodenreider.
The unified medical language system (umls): integrating biomedical terminol- ogy. Nucleic acids research, 32/suppl_) 2004.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1 2020..
Xiongcai Cai, Oscar Perez-Coneha, Enrico Coiera, Fernando Martin-Sanchez, Richard Day, David Roffe, and Blanca Gallego. Real-time prediction of mortality, readmission, and length of stay using electronic health record data. Journal of the American Medical Informatics Association, 23 (3):553-561, 09 2015, ISSN 1067-5027. doi: 10.1093/jamia/ocv11l URL https: //doi. org /tgg/10.0 1093/ hats://mmaaa.........m ocv110.
Chen Chen, Yufei Wang. Bing Li, and Kwok-Yas Lam. Knowledge is flat: A seq2seq generative framework for various knowledge graph completion. arXiv preprint arXiv:2209.07220 2022.
Chen Chen, Yufei Wang, Aixin Sun, Bing Li, and Kwok- Yan Lam. Dipping plms sauce: Bridging structure and text for effective knowledge graph completion via conditional soft prompting. arXiv preprint arXiv:2307.01 709, 2023.
Irene Chen, Monica Agrawal, Steven Horng, and David Sontag. Robustly extracting medical knowledge from ehrs: a case study of learning health knowledge graph. In PACIFIC SYMPO- SIUM ON BIOCOMPUTING 2020, pp. 19-30. World Scientific, 2019.
Edward Choi, Mohammad Taha Bahadori. Andy Schuetz, Walter F Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks, In Machine learning for healthcare conference, pp. 018-318. PMLR, 2016a.
10

---

Published as a conference paper at ICLR 2024
Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Michael Thompson, James Bost, Javier -oooo and Jimeng Sun. Multi-layer representation learning for medical concepts. In proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1404-1504, 2016b.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances neural information processing systems, 29, 2016c.
Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun. Gram: graph-based attention model for healthcare representation learning. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp- 787-795, 2017.
Edward Choi, Cao Xiao, Walter Stewart, and Jimeng Sun. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. Advances in neural information processing systems, 31, 2018.
Edward Choi, Zhen Xu, Yujia Li, Michael Dusenberry, Gerardo Flores, Emily Xue, and Andrew Dai. Learning the graphical structure of electronic health records with graph convolutional transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 606-613, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv: 2014.
Katherine R Courtright, Corey Chivers, Michael Becker, Susan H Regli, Linnea C Pepper, Michael E Draugelis, and Nina R O' Connor. Electronic health record mortality prediction model for targeted palliative care among hospitalized medical patients: pilot quasi-experimenne study. Journal of general internal medicine, 34:1 18:18441-877, 2019.
Kevin Donnelly et al. Snomed-ct: The advanced terminology and coding system for ehealth. Studies in health technology and informatics, 121:279, 2006.
Palmer L Blixhauser A, Steiner C. Clinical classifications software (ccs). 03 2016. URL www.
houp-us ahrq. wwwptooolsoottaaetooooooooeeeoee ccs/ccs. jsp.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric.
arXiv preprint arX:v:1903.024 2019.
Junyi Gao, Cao Xiao, Yasha Wang, Wen Tang, Lucas M Glass, and Jimeng Sun. Stagenet: Stage- neural networks for health risk prediction. In Proceedings of The Web Conference 2020, pp. 530-540, 2020.
Junyi Gao, Chaoqi Yang, Joerg Heintz, Scott Barrows, Elise Albers, Mary Stapel, Sara Warfield Adam Cross, Jimeng Sun, et al. Medml: Fusing medical knowledge and machine learning models for early pediatric covid-19 hospitalization and severity prediction. Iscience, 25(9):104970, 2022.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 1 1462, 2020.
Amelie Gyrard, Manas Gaur, Saeedeh Shekarpour, Krishnaprasad Thirunarayan, and Amit Sheth. Personalized health knowledge graph. In CEUR workshop proceedings, volume 2317. NIH Public Access, 2018.
Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multi- task learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.

---

Published as a conference paper at ICLR 2024
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre- pre-training graph neural networks. arXiv preprint 2019.
Md Shamim Hussain, Mohammed Zaki, and Dharmashankar Subramanian. Global self-attention as replacement for graph convolution. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD '22. pp. 655-665, New York. NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393850. doi: 10. 10.1145/55346688.339999 URL. https:/ / /doi./oo.o////// www/10..//.o. http://1.1114.... / 45346788488 3539296.
Pengcheng Jiang, Shivam Agarwal, Bowen Jin, Xuan Wang, Jimeng Sun, and Jiawei Han. Text augmented open knowledge graph completion via pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 11161-11188 Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/2022 fingggss.inniigs-c....0999 URL https: / htt//aaaaatoolooo/o org/2023. findings acl. 709.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://pyysoone. org/contentminniiii...oaccaaa accesssed August 23, 2021), 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X1v:1412.6980, 2014.
Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and generalization in graph neural networks. Advances in neural information processing systems, 32, 2019.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Re- thinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34:21618-21629, 2021.
Kelsey S Lau-Min, Stephanie Byers Asher, Jessica Chen, Susan M Domchek, Michael Feldman, Steven Joffe, Jeffrey Landgraf, Virginia Speare, Lisa A Varughese, Sony Tuteja, et al. Real-world integration of genomic data into the electronic health record: the pennchart genomics initiative. Genetics in Medicine, 23(4):603-605, 2021.
John Boaz Lee, Ryan Rossi, and Xiangnan Kong. Graph classification using structural attention. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1666-1674, 2018
Scott Levin, Sean Barnes, Matthew Toerper, Amaud Debraine, Anthony DeAngelo, Eric Hamrock, Jeremiah Hinson, Erik Hoyer, Trushar Dungarani, and Eric Howell. Machine-learning based hospital discharge predictions can support multidisciplinary rounds and decrease hospital length- of-stay. BMJ Innovations, 7(2), 2021.
Michelle M Li, Kexin Huang, and Marinka Zitnkk. Graph representation learning in biomedicine and healthcare. Nature Biomedical Engineering, pp. 1-17, 2022.
Yang Li, Buyue Qian, Xianli Zhang, and Hui Liu. Graph neural network-based diagnosis prediction. Big Data, 8(5)337--390, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35, 2023.
Justin Lovelace and Carolyn Rose. A framework for adapting pre-trained language models to knowl- edge graph completion. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5937-5955, 2022.
12

---

Published as a conference paper at ICLR 2024
Chang Lu, Chandan K Reddy, Prithwish Chakraborty, Samantha Kleinberg, and Yue Ning. Col- laborative graph learning with auxiliary text for temporal event prediction in healtheare. arXiv preprint arXiv:2105.07542, 2021a.
Chang Lu, Chandan K Reddy, and Yue Ning. Self-supervised graph learning with hyperbolic em- bedding for temporal health event prediction. IEEE Transactions on Cybernetics, 2021b.
Brady D Lund and Ting Wang. Chatting about chatgpt: how may ai and gpt impact academia and libraries? Library Hi Tech News, 40(3):26 2023.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief- ings in Bioinformaties, 23(6), sep 2022. doi: 10. (0.1093/bibbbbac409 URL BtLhttps: /tt.ooi..org www.1093s2Fbbb2220
Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. Do pre-trained models benefit knowledge graph completion? a reliable evaluation and a reasonable approach. Association for Computational Linguistics, 2022.
Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. Kame: Knowledge-based attention model for diagnosis prediction in healthcare. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 743-752 2018.
Liantao Ma, Junyi Gao, Yasha Wang, Chaohe Zhang, Jiangtao Wang. Wenjie Ruan, Wen Tang, Xin Gao, and Xinyu Ma. Adacare: Explainable clinical health status representation learning via scale- adaptive feature extraction and recalibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 825-832, 2020a.
Liantao Ma, Chaohe Zhang, Yasha Wang, Wenjie Ruan, Jiangtao Wang, Wen Tang, Xinyu Ma, Xin Gao, and Junyi Gao. Concare: Personalized clinical feature embedding via capturing the healthcare context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 833- 840, 2020b.
Riccardo Miotto, Li Li, Brian A Kidd, and Joel T Dudley. Deep patient: an unsupervised represen- tation to predict the future of patients from the electronic health records. Scientific reports, 6(1): 1-10, 2016.
Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv:/109.220 2011.
arXiv preprint
Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, and Svetha Venkatesh. Deepr: convolu- tional net for medical records. IEEE journal of biomedical and health informatics, 21(1):22-30, 2016.
Natalya F Noy, Nigam Shah, Patricia L Whetzel, Benjamin Dai, Michael Dorf, Nicholas Griffith, Clement Jonquet, Daniel L Rubin, Margaret-Anne Storey, Christopher G Chute, et al. Bioportal: ontologies and integrated data resources at the click of a mouse. Nucleic acids research, 37 (suppl 22::22::W WWW7--W772 2009.
OpenAI. Gpt-4 technical report, 2023.
Xuequn Pan and James J Cimino. Locating relevant patient information in electronic health record data using representations of clinical concepts and database structures. In AMIA Annual Sympo- sium Proceedings, volume 2014, pp. 969. American Medical Informatics Association, 2014.
Cecilia Panigutti, Alan Perotti, and Dino Pedreschi. Doctor xai: an ontology-based approach to black-box sequential data classification explanations. In Proceedings of the 2020 conference on ffairness accountability. and transparency, pp- 629-639, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2024
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-learn Machine learning in python. the Journal of machine Learning research, 12:2825-28328 2011.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 1909.01066, 2019.
Peipei Ping, Karol Watson, Jiawei Han, and Alex Bui. Individualized knowledge graph: viable informatics path to precision medicine. Circulation research, 120(7):1078-14)) 1080. 2017.
Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Do- minique Beaini. Recipe for general, powerful, scalable graph transformer. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 14501-14515. Curran Associates, Inc., 2022. URL https::/pprceeeeeee wtwp:eeeeediinnnneeeeeeeeeeeeeee cc httpp/aappeeeaaaalllllaaaaaaaeee walles/paper/2/ file /al.4/66a111111177777777107
Paper Conference pdf.
Nidhi Rastogi and Mohammed J Zaki. Personal health knowledge graphs for patients. arXiv preprint arXiv:2004.00071, 2020.
Maya Rotmensch, Yoni Halpern, Abdulhakim Tlimat, Steven Horn,, and David Sontag. Learning a health knowledge graph from electronic medical records. Scientific reports, 7(1):1-11, 2017.
Junyuan Shang. Tengfei Ma, Cao Xiao, and Jimeng Sun. Pre-training of graph augmented trans- formers for medication recommendation. arXiv preprint arXiv:1906.00346. 2019a.
Junyuan Shang, Cao Xiao, Tengfei Ma, Hongyan Li, and Jimeng Sun. Gamenet: Graph augmented memory networks for recommending medication combination. In proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pp. 1126-1133, 2019b.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. Towards controllable biases in language generation. arXiv preprint arXiv:2005.00268, 2020.
Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. Deep ehr: survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. IEEE journal of biomedical and health informatics, 1285)1159-1 2017.
Sola Shirai, Oshani Seneviratne, and Deborah L McGuinness. Applying personal knowledge graphs to health. arXiv preprint arXiv:2104.0753: 2021.
Chenhao Su, Sheng Gao, and Si Li. Gate: graph-attention augmented temporal neural network for medication recommendation. IEEE Access, 8: 8:125447-1254588 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10900 2017.
Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs. arXiv preprint arXiv:2000.1 1967, 2020a.
Guangtao Wang, Rex Ying, Jing Huang, and Jure Leskovee. Multi-hop attention graph neural net- work. arXiv preprint arXiv: /4332, 2020b.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang. Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint 122.4221,,.0439, 2021.
Cao Xiao, Tengfei Ma. Adji B Dieng, David M Blei, and Fei Wang. Readmission prediction via deep contextual embedding of clinical concepts. PloS one, 13(4):e0195024, 2018.
14

---

Published as a conference paper at ICLR 2024
Xiancheng Xie, Yun Xiong, Philip S Yu, and Yangyong Zhu. Ehr coding with multi-scale feature attention and structured knowledge graph propagation. In Proceedings of the 28th ACM interna- tional conference information and knowledge management, pp. 649 658, 2019.
Chaoqi Yang, Cao Xiao, Lucas Glass, and Jimeng Sun. Change matters: Medication change predic- tion with recurrent residual networks. arXiv preprint DxXiv:210501877: 2021a.
Chaoqi Yang, Cao Xiao, Fenglong Ma, Lucas Glass, and Jimeng Sun. Safedrug: Dual molec- ular graph encoders for recommending effective and safe drug combinations. arXiv preprint arXiv:2105.02711, 2021b.
Chaoqi Yang. Zhenbang Wu, Patrick Jiang, Zhen Lin, Junyi Gao, Benjamin P. Danek, and Jimeng Sun. Pyhealth: A deep learning toolkit for healthcare applications. In Proceedings of the 29th
ACM SIGKDD Conference
Knowledge Discovery and Data Mining. KDD '23, pp. 5788-5789, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9798400701030. doi:
10.1145/3580305.3339 78. URL https: /ooi. org/10 10. 1141/15883305555/ 359911777733333 3599178.
Nianzu Yang. Kaipeng Zeng, Qitian Wu, and Junchi Yan. Moleree: Combinatorial drug recommen- dation with substructure-awann molecular representation learning. In Proceedings of the ACM Web Conference 2023, pp. 4075-4085, 2023b.
Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint 1909v:9909.033
2019.
Changehang Yin, Rongjian Zhao, Buyue Qian, Xin Lv, and Ping Zhang. Domain knowledge guided deep learning with electronic health records. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 738-747. IEEE, 2019.
Bingfeng Zhang, Jimin Xiao, Jianbo Jiao, Yunchao Wei, and Yao Zhao. Affinity attention graph neural network for weakly supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):8082-8066 2021a.
Chaohe Zhang, Xin Gao, Liantao Ma, Yasha Wang. Jiangtao Wang. and Wen Tang. Grasp: generic framework for health status representation learning based on incorporating knowledge from sim- ilar patients. In Proceedings of the AAAI conference on artifiiial intelligence volume 35, pp-
715-723, 2021b.
Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit- Yan Yeung. Gaan: Gated attention networks for learning on large and spatiotemporal graphs. arXiv preprint arXiv:1803. 1803.07294. 2018.
Tianran Zhang, Muhao Chen, and Alex AT Bui. Diagnostic prediction with sequence-of-sets rep- resentation learning for clinical events. In Artificial Intelligence in Medicine: 18th International Conference on Artificial Intelligence in Medicine, AIME 2020, Minneapolis, MN, USA, August 25-28, 2020, Proceedings 18, pp. 348-358 Springer, 2020.
Weicheng Zhu and Narges Razavian. Variationally regularized graph-based representation learn ing for electronic health records. In Proceedings of the Conference on Health, Inference, and Learning. pp. 1-13, 2021.
15

---

Published as a conference paper at ICLR 2024
Contents of Appendix
A Ethics, Limitations, and Risks
17
A.1 Preventing Toxic Behaviors and Ensuring Patient Privacy
17
A.2 Counteracting the Adverse Effects of LLM Hallucinations
18
A.3 Application
18
EHR Dataset Processing
18
C Implementation Details
18
Implementation Details of Step ($3.1): Concept-Specific KG Generation
19
C.2 Implementation Details of Step 2 ($3.2): Personalized KG Composition
19
C33 Implementation Details of Step ($3.3): BAT GNN
19
C,4 Experiment Environments
20
C,5 Training Details
20
D Knowledge Graph Construction
20
D.I Prompting KG from Large Language Model
20
D.2 Sampling Subgraph from Existing KG
22
D.3 Knowledge Graphs after Clustering
D.4 Analysis on GPT-UMLS-KG
23
"Patient As Graph" and "Patient As a Node"
24
E. Patient Representation Learning.
25
Importance Score
25
Hyper-parameter Tuning
25
G.1 Hyper-parameters for Clustering
26
G.2 Hyper-parameters for the Bi-attention GNN
27
H Notation Table
28
16

---

Published as a conference paper at ICLR 2024
A ETHICS, LIMITATIONS, AND RISKS
In this study, we introduce novel framework, GRAPHCARE, which generates knowledge graphs (KGs) by leveraging relational knowledge from large language models (LLMs) and extracting infor- mation from existing KGs. This methodology is designed to provide an advanced tool for healthcare prediction tasks, enhancing their accuracy and interpretability. However, the ethical considerations associated with our approach warrant careful attention. Previous research has shown that LLMs may encode biases related to race, gender, and other demographic attributes (Sheng et al.. 2020; Weidinger et al., 2021). Furthermore, they may potentially generate toxic outputs (Gehman al., 2020). Such biases and toxicity could inadvertently influence the content of the knowledge graphs generated by our proposed GRAPHCARE, which relies on these LLMs for information extraction. Furthermore, the issue of privacy has emerged as paramount concern associated with LLM usage (Lund & Wang. 2023).
We explain the limitations of GRAPHCARE and describe the measures we have implemented to counteract or mitigate these ethical concerns as follows.
A.1 PREVENTING TOXIC BEHAVIORS AND ENSURING PATIENT PRIVACY
Primarily, the LLM within GRAPHCARE is exclusively utilized extract knowledge associated with medical concepts. This focused usage drastically reduces the chances of inheriting wider so- cial biases manifesting toxic behaviors intrinsic the parent LLMs. Furthermore, we ensure that no patient data is introduced into any open-source software. This measure fortifies patient con- fidentiality and negates the possibility of injecting individual biases into the knowledge graphs. This commitment is further elucidated in Figure 5.
Data Preparation Stage (no patient data used)
Conditions
Procedures
Drugs (e ATC)
[e.g. ICD9CM)
(e.g. ICDSPROC)
Medical "fever
"oough"
"injection"
"Duppedee""
concepts
LLM Prompting
Concept-
wise Graph
Knowledge graph is stored for each medical concept
Local Model Training Stage (no LLM used)
Compose the patient's jersenplized knowledge graph
Query the graph storage with medieal coneepts
Training
Patient Data
Figure 5: High-level View of GRAPHCARE for Clarification Ethical Considerations. GRAPHCARE consists two general stages: data preparation and local model training. During data preparation, LLM solely extracts knowledge graphs associated with medical concepts, without accessing any patient's 's data. At the local model training stage, personalized knowledge graphs for patients are constructed using the knowledge graphs corresponding medical concepts found in the patient's EHR, without any engagement of the LLM. A local graph storage serves both the repository for the procured medical concept-wise KGs and the mechanism for querying KGs for personalized KG compositions.
17

---

Published as a conference paper at ICLR 2024
A.2 COUNTERACTING THE ADVERSE EFFECTS OF LLM HALLUCINATIONS
The predictive efficacy of GRAPHCARE is intrinsically tied the veracity of knowledge graph triples sourced from LLMs. Hence, hallucinations within LLMs can detrimentally skew the per- formance of the model. To counterbalance this, we collaborate with medical professional (MD) to scrutinize the accuracy of LLM-derived triples and expunged content that might be detrimen- tal (further details are provided in Appendix D.I), Leveraging domain expert knowledge on triple evaluation and selection greatly minimizes the negative impacts of LLM hallucinations, ensuring high-quality knowledge probing from LLM.
A.3 APPLICATION
It's important to emphasize that GRAPHCARE is primarily intended for research purposes. This means that while it offers insights and can provide valuable information, it has not been certified or endorsed for clinical or diagnostic use. Any implementation or interpretation of GRAPHCARE should be undertaken with the clear understanding of its experimental nature.
While GRAPHCARE serves as an advanced tool for healthcare prediction tasks, it should not replace or undermine the expertise of medical professionals. We strongly advise against relying solely on its predictions for healthcare decisions. Medical doctors possess extensive training and clinical experience, and their judgment should always prioritized over automated systems. Patients and healthcare providers should use the information from GRAPHCARE as supplementary and should always consult with healthcare professionals before making any medical decisions.
B EHR DATASET PROCESSING
In this paper,
MIMIC-III and MIMIC-IV datasets. Both datasets under PhysioNer Cre- dentialed Health Data License 1.5.0² We employ PyHealth (Yang et al., 2023a) to process these two datasets. PyHealth has EHR dataset pre-processing pipeline that standardizes the datasets, organizing each patient's ' data into several visits, where each visit contains unique and specified fea- ture lists. For our experiments, we create feature lists composed of conditions and procedures for Length of Stay (LOS) prediction and drug recommendations. For the prediction tasks of mortality and readmission, we include the medication (drug) list addition to the condition and procedure lists.
Subsequent to the parsing of the datasets, PyHealth also enables the mapping of medical concepts across various coding systems using the provided code maps. The involved coding systems in this process are ICD-9*, ICD-104.5, CCS$ NDC and ATC our experiment, we convert 1 11,736 ICD- 9-CM codes and 72,446 ICD-10-CE codes into 285 CCS-CM codes to capture condition concepts. Similarly, we map 4,670 ICD-9- 9-PRCC codes (a part of ICD-9-CM for procedure coding) and 79,758 ICD-10-PCS codes 231 CCS-PROC codes for procedure concepts. For drug concepts, convert 1,143,020 NDC codes into 269 level-3 ATC codes. This mapping process enhances the training speed and predictive performance of the model by reducing the granularity of medical concepts.
C IMPLEMENTATION DETAILS
In this section, we present the implementation details of GRAPHCARE, aligning it closely with the methodology described in Section 3, which improves reproducibility and clarity.
'https: httpss//wwhhoooggggaggrg//////as
18

---

Published as a conference paper at ICLR 2024
C.I IMPLEMENTATION DETAILS OF STEP ($3.1): CONCEPT-SPECIFC KG GENERATION
Medical concepts C, and d. After the EHR data preprocessing illustrated Appendix B, we have 285 conditions (cc] 285), 231 procedures ((pl = 231), and 269 drugs (ld) 269).
LLM-based KG extraction. We detail this process in Appendix D.I.
Subgraph sampling from existing KGs. We detail this process in Appendix D.2.
The hyperparameter studies regarding (1) LLM prompting times X, and (2) x. hops from the source entity showcased in Table 6, and 8.
Node and Edge Clustering. To analyze the global graph G. we obtain the word embeddings for each node and edge. These embeddings have 1536 dimensions and are sourced from the second- generation GPT-3 model, specifically the text eeteeemmmeeiing beobeddiin---dde ada--dda-a--- single vector embedding of the input text regardless of the number of tokens it contains. We Scikit-learn 1.2.1 (Pedregosa et al., 2011) to implement agglomerative clustering. We detail the hyperparameter study (for distance threshold 8) of clustering Appendix G.I. 0.15 was chosen based on the study.
The model will output
C.2 IMPLEMENTATION DETAILS OF STEP ($3.2): PERSONALIZED KG COMPOSITION
Visit Processing: Iterate through each visit in the patient's EHR. For each visit visity, we have the medical concepts (concept,,, concett,...)
Inter-Visit Relationships:
Identify and establish connections between nodes across different visits if they share relationship in the global graph G'. These connections represented by the set Einer, highlighting the continuity and progression in the patient's medical history.
Final Personalized KG Assembly: Combine all the integrated concept-specifie KGs, the patient node P. and the inter-visit relationships to form the final personalized KG for the patient, Gat(i)- This KG encapsulates the entire medical history of the patient, structured in a cohesive and inter- connected manner.
C.3 IMPLEMENTATION DETAILS OF STEP ($3.3): BAT GNN
Attention Initialization. Keywords we experimented for attention initialization are in Table 4
Table 4: Keyword candidates we attempted for attention initialization. We highlight the keywords we finally used in the experiments.
Task
Conditions
Procedures
Drugs
MT.
terminal condition, critical diagnosis, end-stage, life-threatening
critical interventions, life-saving measures, resuscitation, emergency procedure
palliative medication. end-of of-Tif drugs. life support drugs, emergency meds
RA. chronic ailment, postoperative complication, recurrent, readmission-prone
follow-up procedure, secendary intervention, post-treatment, treatment review
medication. postoperative drugs. treatment continuation, follow-up meds
LOS acute condition. severe diagnosis, long long-term ailment, extended-care diagnosis
major surgery. intensive procedure, long recovery intervention, extended hospitalization
Drug.
chronic disease, acute diagnosis, symptomatic, treatable condition
diagnostic procedure. treatment procedure, medical intervention. innggindicative procedure
hhttps://opends http:///annaaacoooo//oo///////// and- peddd-ippoveddeeeeeee -ngeeedddi---ddee
19

---

Published as a conference paper at ICLR 2024
Pateint Representations Study. We detail this in Appendix E.1. Based the results, joint representation in experiments.
Hyperparameter Study. We detail this in Appendix G.2. C.4 EXPERIMENT ENVIRONMENTS
Hardware. All experiments conducted machine equipped with AMD EPYC 7513 32-Core Processors, 528GB RAM, eight NVIDIA RTX A6000 GPUs, and CUDA 11.7.
Software. We implement GRAPICARE using Python 3.8. 13. PyTorch 1.12.0 (Paszke al., 2019), and PyTorch Geometric 2.3.0 (Fey & Lenssen, 2019). We employ PyHealth (Yang et al., 2023a) to pre-process the EHR data (illustrated in Appendix B). We utilize medical code mappings from ICD. (9/10) to CCS for conditions and procedures, from NDC to ATC (level-3) for drugs. The mapping files are provided by AHRQ (Elixhauser A, 2016) and BioPortal (Noy al., 2009). We use Gephi (Bastian et al., 2009) for knowledge graph visualization.
C.5 TRAINING DETAILS
General Setting. We split the dataset by 8:1:1 for training/validaaion//esnnng data, and we use Adam (Kingma & Ba, 2014) the optimizer. Based on our hyperparameter study in Appendix G.2, we set learning rate 1e-5, weight decay le-5, batch size 4, and hidden dimension 128. All models are trained via 50 epochs over all patient samples, and the early stopping strategy monitored by AUROC with 10 epochs is applied.
Features for Different Tasks. We take conditions and procedures as the features for the length- of-stay prediction and drug recommendation and additionally take drugs as features for mortality prediction and readmission prediction.
Baseline Models. We use PyHealth (Yang et al., 2023a) pipeline to load the implemented models with their best reported settings. For GRAPHCARE w/ GPS (Rampášek et al. (2022)), we apply LapPE (Kreuzer et al. (2021)) as the Laplacian positional encoding, GINE (Hu et al. (2019)) as the local message-passing mechanism, and Transformer (Vaswani et al. (2017)) for the global attention.
D KNOWLEDGE GRAPH CONSTRUCTION
In this section, we illustrate our solution to construct a biomedical knowledge graph (KG) for each medical concept by prompting from large language model (LLM) and sampling subgraph from a well-established KG.
D. PROMPTING KG FROM LARGE LANGUAGE MODEL
GPT-KG. Figure 6 showcases carefully designed prompt for the retrieval of biomedical KG from a generative LLM. The main goal of this approach is to leverage the extensive knowledge embedded in the LLM extract meaningful triples consisting of two entities and relationship.
In our strategy, we begin with a prompt related to a medical condition, a procedure, or a drug- The LLM then tasked with generating list of updates that extrapolate as many relation- ships as possible from this prompt. Each update is a triple following the format [ENTITY 1, RELATIONSHIP, ENTITY 2 where ENTITY 1 and ENTITY should be nouns. Our goal is to generate these triples in both breadth (a wide variety of entities and relationships related to the initial term) and depth (following chains relationships to discover new entities and relationships). The process continues until we obtain a list with 100 KG triples. This prompting-basee approach provides a structured, interconnected knowledge graph from the unstructured knowledge embedded in the LLM, which proves especially beneficial for personalized KG generation.
In the experiment, we iterate through the vocabulary of conditions, procedures, and drugs contained
in CCS and ATC (level-3) with their code-name mappings M : e + where eA is the
Phttps://www.hhr http://www.ccuuuuue haapppuuuhhhr....... gtv/g.gooooooooo "httpss//bioportaal bi bioontol ogy . org/ wtt.:///oooooggie/eeeaato.o.
20

---

Published as a conference paper at ICLR 2024
def ehr for__g__rompptng term, category):
category "condition":
example
Example:
prompt: systemic lupus erythematosus
updates: [[systemic lupus erythematosus, is an, autoimmune condition), [systemic lupus erythematosus. may cause, nephritis], [anti-nuclear antigen, test for, systemic lupus erythematosus), [systemic lupus erythematosus, is treated with, steroids], Imetthylpreenisollnne, is steroid]]
elif category = "procedure":
example =
Example:
prompt: endoscopy
updates: [[endoscopy, is medical procedurel, Lendoscopy, used for, diagnosis), lendoscopic biopsy, is type of, endoscopy), [endoscopic biopsy, can detect, ulcers]]
elif category "drug": example
Example:
prompt: iobenzamic acid
updates: [[iobenzamic acid, is a, drugl, [iobenzamic acid, may have, side effetts) [side effects, can include, nausea], [iobenzamic acid, used as, X-ray contrast agent], [iobenzamic acid, formula, C16H1313N203]]
gpt GPT() response gpt.chat((
funn
Given prompt (a medical condition/ ponditiioo/rocedrr///ruug)) extrapo late as many relationships possible of it and provide list of updates.
The relationships should be helpful for healthcare prediction (e.g., drug recommendation, mortality prediction, readmission prediction ...) Each update should be exactly in format of (ENTITY 1, RELATIONSHIP, ENTITY 21. The relationship is directed, so the order matters. Both ENTITY and ENTITY should be noun.
Any element [ENTITY 1, RELATIONSHIP, ENTITY should conclusive, make it as short as possible.
Do this in both breadth and depth. Expand [ENTITY 1, RELATIONSHIP, ENTITY 2] until the size reaches 100.
{example}
prompt: {term)
updates:
Process
response triples
triples parse (respons))))
return triples
Figure 6: Prompting knowledge graphs for medical concepts (EHR terms) from GPT.
21

---

Published as a conference paper at ICLR 2024
corresponding name for the medical code e. For each term e^, input it with its category (either "condition". "procedure" or "drug") to the function ehr_kg_prompt times and compose the graphs of all runs into one, i..., G = (Gi U G5 U U Gx), ) to obtain more comprehensive graphs.
() shown Figure X
Table 5: Expert Evaluation of Knowledge Graph Triples for Medical Concepts. We assess the quality of triples for randomly selected 50 medical concepts from each coding system (vocabulary). Four metrics: breadth, depth, faithfulness, and bias used for the evaluation, each scored a 1-5 scale. A higher score in- dicates better performance. Breadth signifies the variety of triples in which the target medical concept features as an entity. Depth represents degree interconnectedness triples (e.g., given triple (é1, Fi, e2) and to : (e2, 72, ea), t2 is an extension of (1), Faithfulness quantifies the overall factual accuracy of the triples. We present both the average score and standard deviation for each metric in our evaluation.
Concept type
Conditions
Procedures
Drugs
Concept type Conditions
Procedures CCSPROC
Drugs
Vocabulary
CCSCM
CCSPROC
ATC-3
Vocabulary
CCSCM
ATC-3
Breadth
4.2 0.4
3.8 0.3
4.0 + 0.2
Breadth
4.6 0.2
4.3
4.6 + 0.2
Depth
4.0 0.3
3.2
3.3 0.4
Depth
3.8 0.3
3.9
4.1 0.4
Faithfulness
4.5 + 0.3
4.3
4.5
Faithfulness
4.8 0.1
4.9
4.6
(a) Evaluation of GPT-KG.
(b) Evaluation of GPT-UMLS-KGG
We engaged a medical professional collaborating with us to evaluate the KG triples produced by LLM. The outcomes of this evaluation are presented in Table 5a. As evidenced by the results, the triples generated by GPT-4 exhibit high quality in terms their breadth, depth, and faithfulness.
Furthermore, after clustering of nodes / edges with 0.15, we futher eliminated 27 out of the 4,626 nodes (clusters) due to their inclusion of inaccurate or potential misleading content, with the help from medical professionals. This measure resulted in the removal of 3,393 KG triples, addressing potential ethical concerns. We also asked medical professionals for their help to remove triples that contain inaccurate, biased, misleading information, which resulted in the removal of 4,539 triples. This triple filtering process addresses the potential echical concerns.
As result, obtained 65,993 non-redundant KG triples with 48,914 unique entities and 8,067 unique relations when we set = 3, as shown in Table 6. For future work, we will explore to use this prompting-ben method construct more task-specific KGs, aiming at providing more relevant triples especially beneficial to certain prediction.
D.2 SAMPLING SUBGRAPH FROM EXISTING KG
UMLS-KG. To extract subgraphs for medical concepts from existing well-established biomedical KG like UMLS (Bodenreider, 2004), we take the following steps:
1. We text-embeddiig--adee the UMLS KG and all concepts contained in the target medical coding system (CCS-CM, CCS-PROC in case) for conditions and procedures. For drugs (ATC-3), use the existing ATC-to-UMLS. CCU mapping provided by BioPortal
to retrieve the word embedding of all entities in
2. For each medical concept, we search the entity in UMLS that with the most similar word embeeding, and create mapping from CCS/ATC concept names to those entities.
3. For each UMLS entity in this mapping, we apply subgraph sampling described in the fol- lowing Algorithm 1.
In Algorithm 1. hop limit i and window size c. brief, we search all the triples containing e for the first hop and search triples containing the other entity for each previous-hop triple. When setting к 2 and e = 5, we obtain 265, 265,587 non-redundant KG triples with 137,845 unique entities and 94 unique relations, shown in Table 6.
We have four arguments- medical concept (in CCS/ATC), source KG & (UMLS),
Phttps://bigge bioontol ogy. org/ ontolgiie
22

---

Published as a conference paper at ICLR 2024
Algorithm Subgraph Sampling
1: procedure
concept e, KG g. hop limit K, window size c)
2: Initialize an empty list Q and an empty graph G (Ve &
sub(x)
sub(r): Sub(x))
3: Add e to Q
4: V Van(()) V snb(ic) U {e}
5: for i to do
6:
Initialize an empty list Quext for all ent E Q do if then Retrieve all triples (ent, rel, ent') or (ent', rel, ent) from g
7:
8:
9:
10:
else
11:
Randomly retrieve triples (ent, rel, ent') or (ent', rel, ent) from
12:
end if
13:
Add retrieved triples to sub((s)
14:
Ve sub(k) + V Sub(e)) U {ent'}
Add ent' to Quext
end for 16:
Q - Quext end for
18:
return G (ab((k) end procedure
20:
Table 6: Statistics of GPT-KG (generated through prompting $D.1) and UMLS-KG (extracted through subgraph sampling $D.2). We report the data in the format of (#f unique nodes, # unique edges, # triples).
KG
Hyperpurameter
Condition
Procedure
Drug
Total
GPT-KG
x=3
(17780, 3633, 22421)
(9636, 1991, 10429)
(26922. 4362. 33380) (6509, 7547)
(48914, 8067, 65993) (20466, 66, 29334)
UMLS-KG
(11895, 17747)
(3614, 4158)
UMLS-KG
2=2. 6
(86143, 70, 151294)
(68129, 71, 98817)
(63274. 79, 87267)
(137845, 94, 265587)
GPT-UMLS-KG. By integrating concept-specific KGs produced by GPT-4 with those from UMLS, we constructed GPT-UMI The expert assessment of this amalgamated KG is presented in Table 5b. Notably, compared to GPT-KG, there is an enhancement in quality across all dimensions, consistent with our observations in Figure
D.3 KNOWLEDGE GRAPHS AFTER CLUSTERING
Table 7: Statistics of GPT-KG, UMLS-KG, and GPT-UMLS-KG after node/edge clustering.
KG
Hyperparameter
Nodes
# Edges
# #Triples
GPT-KG
3=
4599
752
31325
UMLS-KG
3053
12421
UMLS-KG
x=2,
10805
81073
GPT-UMLS-KG
X=3, k=
6355
774
40496
GPT-UMLS-KG
x=3, x=2, c
12284
785
104460
In Table 7, we present the KGs following the node/edge clustering process detailed in $3.1, with set value of 8 0.15 (as optimized in Appendix G.1). We note a notably low triple union between GPT-KG and UML.-KG. This suggests that the knowledge from one can significantly complement the other. Consequently, GPT-UML.S-KE is poised outperform either of the two individual KGs. This inference is empirically supported by our results displayed in Figure
D.4 ANALYSIS ON GPT-UMLS-KG
Table illustrates the impact of the two GPT-UMLS-EE variants on enhancing the performance of EHR predictions. It is evident that the performance significantly improves when 1, while
23

---

Published as a conference paper at ICLR 2024
Table 8: Comparison of the performance gain from the GPT-UMLS-KE with 1-hop and 2-hop concept-specific subgraph sampled from UMLS.
MIMIC-III
MIMIC-IV
KG
MT.
RA.
LOS
Drug.
MT.
RA.
LOS
Drug.
GPT-UMLS-KG (x=3, x=l)
70.3
69.7
37.5
66.8
73.1
68.5
34.2
63.9
GPT-UMLS-KG (x=3, x=2, c
68.4
67.2
35.4
63.4
72.6
66.7
33.4
62.2
the performance with k sometimes even fails to surpass the baseline performance without any external knowledge (e.g. outperformed by RETAIN on MIMIC-III drug recommendation task), as we compare the performance to Table 2. Possible explanations for this outcome:
The constrained window size (c) increases the randomness of the triples sampled after the initial 1-hop, resulting in a proliferation isolated nodes and the formation of isolated clusters. This situation poses considerable challenge for the GRAPHCARE model to ef- fectively learn from the knowledge graph.
The increased randomness is very likely to exclude critical triples originating from a source node, leading the propagation of irrelevant knowledge triples (noise) that ultimately detrimentally affect the model's performance.
effective method to sample more useful triples from existing KGs becomes one of our future works.
Therefore, developing
E
"PATIENT AS GRAPH" AND "PATIENT As A NODE"
Patient as Graph
Patient as Node
Gyal(s)
22
T2
T'
Pi
V.L.3
FE.1.
A312
Figure 7: Comparison of two patient representations in GRAPHCARE. Left: patient as graph covering the information in all nodes. Right: patient node only connected the nodes of the direct medical codes (the larger ones) recorded in the EHR dataset. x, denotes the j-th for patient v.y.k denotes the k-th node of j-th visit for patient i. The connections among nodes are either mistr or across-visit
Figure presents different patient representations. When viewed as graph, the patient rep- resentation aims to encapsulate a comprehensive summary of all nodes, thus providing a broad overview of information. However, this approach may also include more noise due its extensive scope. In contrast, when patient is represented directly corresponding EHR nodes. This approach ensures a precise match with the patient's EHR data, offering a more accurate, albeit narrower, representation. Although this method provides more focused insight, it also inevitably discards information from other nodes, thus potentially los- ing broader contextual data. Therefore, the choice between these two representations hinges the balance between precision and the extent of information required. In our experiment, we introduce joint embedding composed by concatenating those two embeddings, balanced solution.
node, the information is aggregated solely from
24

---

Published as a conference paper at ICLR 2024
Prediction
Prediction
Recommandaaiin
0.500
0.523
0.79
0.445
0,6466
6.77
0.445
0.484
Mode
Mode
Hods
Figure 8: Performance of healthcare predictions with three types of patient representations ($3.3): (1) graph patient graph embedding obtained through mean pooling node embedding: (2) node patient node embedding connected to the direct EHR node; (3) joint embedding concatenated by (1) and (2). We use GPT-KG to perform this analysis.
E. PATIENT REPRESENTATION LEARNING.
We further discuss the performance of different patient representations in GRAPHCARE, depicted in Figure 8. We calculate the average over 20 independent runs for each type of patient representation and for each task. Our observations reveal that the patient node embedding presents more stability as it is computed by averaging the direct EHR nodes. These nodes are rich in precise information, thereby reducing noise, but they offer limited global information across the graph. On the other hand, patient graph embedding consistently exhibits the most significant variance, with the largest distance observed between the maximum and minimum outliers. Despite capturing a broader scope of information, the graph embedding performs less effectively due to the increased noise. This is attributed to its derivation method that averages all node embeddings within a patient's personalized KG, inherently incorporating more diverse and complex set of information. The joint embedding operates as a balanced compromise between the node and graph embeddings. It allows GRAPHCARE to learn from both local and global information. Despite the increased noise, the joint embedding provides an enriched context that improves the model's understanding and prediction capabilities.
F IMPORTANCE SCORE
To provide insights into the GRAPHCARE'' decision-making process, we propose an interpretation method that computes the importance scores for the entities and relationships in the personalized knowledge graph. We first compute the entity importance scores as the sum of the product of node- level attention weights aij. and visit-level attention weights Brg (obtained by Eq (3)) over all visits, (1) and relationship importance scores as the edge weights w (k.k')) summed over all GNN layers:
L-1
Tent 4,
ΣΣ
THI K.K.K'
Σ
W ( WK(K
where Jent is the importance score of entity h and Tel k is the importance score of the relation- ship between entities & and k. To identify the most crucial entities and relationships, we can also compute the top K entitics and relationships with the highest importance scores, denoted as s in descending order of their importance:
Th = {s |
1 1
19 ((K) T.sk = {8| & I rel (((K))
where Iem (,(K) and pel (.(K) are the K-th highest importance scores for entities and relationships, re- spectively, T and 7,d represent the top K entities and relationships for patient i, respectively. By analyzing the top entities and relationships, we can gain a better understanding of the model's decision-makin process and identify the most influential factors in its predictions.
G HYPER-PARAMETER TUNING
Given that GRAPHCARE utilizes personalized knowledge graphs (KGs) as inputs for health- care predictions, the representativeness of the constructed graphs becomes critical in the prediction process. The quality and structure of these KGs can significantly influence the performance of
25

---

Published as a conference paper at ICLR 2024
predictive model, underlining the importance of thoroughly investigating the hyperparameters involved in their construction and subsequent analysis via the BAT GNN model. Therefore, we meticulously examine both the hyperparameters for KG node/edge clustering and those for our pro- posed BAT Graph Neural Network (GNN) model. We use GPT-KG as the external knowledge and use validation set of EHR data for more efficient parameter searching.
G.1 PAPER-RARAMETER FOR CLUSTERING
Table 9: Clustering hyperparameter tuning. Tested on GPT-KG.
Mortality AUPRC
Readmission
Threshold
# Cluster
AUROC
AUPRC
AUROC
0.05
29681
12.2
61.3
65.5
63.5
0.1
14662
13.3
65.2
70.0
67.4
0.15
4599
15.7
69.6
72.6
68.9
0.2
883
13.9
67.8
67.8
66.7
LOS AUROC
Drug Rec.
Threshold 6
Cluster
Fl-score
Fl-score
AUROC
0.05
15094
32.8
77.4
62.1
94.2
0.1
7941
34.7
79.7
64.8
94.5
0.15
2755
36.6
80.2
65.2
95.1
0.2
589
34.1
77.9
63.8
94.3
5 0.05
8 0.13
8 = 0.20
eomple 1) "Fever and night sweats")
Exsepte 2: "brakn tunors":
6 0.06
fever and night quests, nighe wuests
bratn tuaens, brain tume,, brain EanCK"
(5lxe12)
(sizec3)
fever and night smaats, night sweats, awest test, hysartidrosis hypohidrosis, detreased smaating, bypertidrosiss treat channdere, treet custiig ctrendere, trear hyperhidestt,, hest and wuesting, cwetting, diarrhas mccrextoe eveating
brain tueors, brain turer glsom, varingtons, clipodndrogiion.. brais cancer, neree tueor, nptrobiaston, metillebissttmn, suurttitamattssss developing zrain tumera, opardymmma, brain cancer originating azenomaal calls, trais cancer originating in shaped cails, brain catter originating oligodendrsyysaa, tumers in the nervoss spstem, marsoblatoon
0.15
(51c0:36)
(s1me:33)
glisblastoms
Froe night cueste, night wests, acceestus weicture, dry hialdity crai dryness, cold hualdity, cold str and tumlstty tustet recation, nest test, trying sut, dry eye syettse, combines with hartsitter, craeting, dry mots averaned, apantsrants, estemure Fiem, dry wuth and thiss, the skin dry, dr eucch and theot, decesssed serecing, typeriarosi) serestonia,
brain tumes, bone matuotasia sarcene, dectal carcinena
trauformition, provention uterare froe gestrointtsiinnl stronel tunors (01575)) secondary milignerey treutment, riss fectors ling cancer, normalignan conditions,
sore netestanis
eauth
netsture,
= 0.20
0.20
assessmant, benigg sueplage of stanss, noncancroos groeth, tchuenrea,, cance: gattents circtoure, urster canter, maligaare
(alce:50)
treatrerts, heiddifier used for dry eye treeswest, escessive swoating, retain solsture, dry skin, throt dryness, reduce dry nouth symproms, drynes, dry inhidesiss sisenders,
(2120:524)
treaments, dreoling, dryness, swatting swattig
speead of cancer to tones, risk of indorstrial rancer, treat breast cancar, are rance", cartain canter trastannss
sisordera, treat typortidrcass, dry syes, treat ay eoath, heat mesting, ary the skin, somating, to prevent socass molsture siarthea accessive smaating, hunidity tunperature
breast prevention, detection ovariat cueors, cancor eanggoont
Figure 9: Comparison between the node clustering over GPT-KG and UMLS-KG. Above: we random sample 80 clusters with each distance threshold 6 applied. Each figure visually represents the clustering of words, with color consistency denoting membership to the same cluster. Below: we provide two examples of the clusters with different 8's for the given words ("fever and night sweats" and "brain tumors").
Table 9 presents the performance of GRAPHCARE across four tasks on the MIMIC-III dataset, with varying agglomerative clustering distance thresholds 5 E (0.05, 0. 0.15, 0.2). We evaluate the
26

---

Published as a conference paper at ICLR 2024
performance with the GPT-KG. The results reveal that the model achieves optimal performance when 8 = 0.15. This outcome can be attributed to the following reasons: when 6 is small, nodes of high similarity may be incorrectly classified distinct, complicating the learning process for the model. Conversely, if 5 is large, dissimilar nodes could be inaccurately clustered together, which further challenges the training process. Examples in Figure 9 further demonstrate our findings.
The examples presented Figure illustrate the significant influence of the distance threshold on the semantic coherence of the clusters. When 8 0.20, the clusters tend to incorporate words that aren't strongly semantically related to the given word. For instance, "humidity" is inappropriately grouped with "fever and night sweats", and "breast cancer" is incorrectly associated with "brain tumors". Conversely, when 8 = 0.05, the restrictive threshold fails to capture several words closely related the given word, such as the absence of "heat and sweating" the cluster for "fever and night sweats", and "glioma" for "brain tumors".
Striking an optimal balance, when 5 0.15, the resulting clusters exhibit a desirable semantic coherence. Most words within these clusters are meaningfully related to the given word. This observation underlines the importance of selecting an appropriate 8 value to ensure the extraction of semantically consistent and comprehensive clusters. This is pivotal step, the quality of these clusters has a direct impact on subsequent healthcare prediction tasks, which rely on the KG constructed through this process.
G.2 HYPER-PARAMETER FOR THE BI-ATTENTION GNN
Batch
Hidden Dimension
Leaminn Rate
1.0
1.0
1.0
0.5
0.5
500me
0.6
0.2
0.2
0.2
26
128
255
512
16:05
0,0001
0.002
Value
Value
Weight Decay
*
*
0.8
0.5
0.8
Store
Score
0.4
0.2
0.2
0.2-
0.0001
0.001
0.08
1.0
Mortality
Readmission
LOS
Drug Recommendation
Figure 10: Hyper-paras parameter tuning. We tune each parameter while keeping other hyperparameters fixed as their default values (batch size: 4, hidden dimension: 128, learning rate: le-5, weight decay: 1e-5, decay rate: 0.01, layers: 1). Score denotes the normalized AUROC in the range of [O, 1]. which shows the relative performance of a specific setting compared to others. The best value each hyperparameter for each task labeled as a star.
Dimension
Learning
Weight Decay
Decay Rate
Layers
Mortality
128
1c-5
lc-5
0.01
Readmission
128
le:5
le-5
0.01
Length Stay (L.OS)
128
le:5
le.5
0.03
Drug Recommendation
128
le-5
le-5
0.03
3
Table 10: Hyper-parameter for the BAT GNN model for different tasks.
For our proposed BAT GNN we tune the following aoper-parameters: batch size in (4, 16, 32, 64), hidden dimension in (128, 256, 512), learning in (1e-3, le-4, le-5), weight decay in {le-3,
27

---

Published as a conference paper at ICLR 2024
le-4, le-5), decay rate in {0.01, 0.02, 0.03, 0.04) and number of layers L in {1, 2, 3, 4). We employed throughout the experiments
show the tuning detail in Figure 10. The presented in this paper are consolidated and presented in Table 10. For the sake of maintaining more fair and balanced comparison, we align the batch size, hidden dimension, learning rate, and weight decay with those of the baseline models.
H NOTATION TABLE
For clarity, we have attached notation table here, describing all symbols used in the main paper.
Table 11: Notations and Descriptions in GRAPHCARE
Notation
Description
Notations in Step 1 ($3.1)
E {c. d}
A medical concept in [conditions, procedures, drugs) Sizes of sets of medical concepts
Jc). pp.
GO VE
KG, nodes, and edges for each medical concept
GLMMO) (IMON) LIMOO
KG for each medical concept, obtained through prompting LLM x times
Ve
Nodes and edges the KG obtained through prompting LLM times (Appendix D.I)
V sulde) subleg: Sab(ss) Cy.Ce
r-hopp subgraph for concept, obtained thourgh subgraph sampling (A (Appendix D.2)
Nodes and edges the x-hop subgraph obtained thourgh subgraph sampling Clustering mappings for nodes edges
Distance threshold cosine similarity
G. V.E G.V.e.
Global graph composed by all concept-specific KGs, nodes and edges
New global graph, nodes and edges after clustering
W
Dimension of the word embedding (Initial) node and edge embeddings
H' H
Notations Step 2 (83.2)
P
Patient node
Gpst, Spat:Epa
Personalized KG for a patient
Edge connecting patient node and node of medical concepts patient's EHR Visit-subgraph
G
j-th visit the patient The node the j-th visit the patient
Pig. (Bijik + (
The edge between node & and 0r A Interconnected edges across visit-suhgraphs
Emter
Notations in Step 3 ($3.3)
(++1)
Updated node representation node k (I 1)-th layer of GNN Activation function
W
Learnable weight matrix at I-th layer
Function to aggregate node representations
b
Bias vector at 1-th layer Neighbors node k Hidden embedding of k-th node in j-th visit-subgraph of patient Hidden embedding of edge hetween nodes 14 and ti.""" Learnable matrices in R earnable vectors in R99
bagik
W W.
b b,
Input emheddings of the node and edge Size of hidden embedding
9
a, BL,
Node-Jevel and visit-level attention weights
Bigi G.
Multi-hot vector and matrix for visit-subgruph patient's graph Learnable parameters for node-level attention visit-level attention Bias vectors for node-level attention visit-level attention Decay coefficient vector Decay rate
W.. wa
b. by
X
T
Wir
Word embedding of keyword task-feature pair Computed weight for mith node global graph G'
Wo
h (L)
Node embeddings of final layer for predictions
Putient graph embedding
Patient node emhedding
IA
Binary label indicating direct medical concept
graph podl
Logits from different embeddings after MI.P
28


Reviews:
[{"review_id": "peTavYSO0k", "reviewer": "Reviewer_ZFn4", "summary": "The paper introduces GraphCare, a framework that utilizes external knowledge graphs to enhance healthcare predictions and decision-making. This framework generates personalized knowledge graphs for patients, thus improving accuracy and performance in crucial healthcare prediction tasks. The methodology of GraphCare, including its process of extracting knowledge from large language models and external biomedical KGs, is discussed in detail. Moreover, the paper presents experimental results on the MIMIC-III and MIMIC-IV datasets, demonstrating the framework's effectiveness in enhancing prediction accuracy and explainability. Finally, the limitations and potential future directions of GraphCare are discussed.", "questions": "- The authors are encouraged to provide additional results on the quality assurance of their constructed personalized knowledge graphs. The significance of this is underscored by the direct impact these graphs have on the results.\n- The authors should also address the concerns highlighted in the 'weaknesses' section of the review.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- GraphCare generates personalized knowledge graphs for patients, enhancing the accuracy of predictions. This insight is compelling, as personalized predictions are often more accurate than general ones.\n- The paper presents experimental results on the MIMIC-III and MIMIC-IV datasets, which demonstrate the framework's effectiveness in improving prediction accuracy and explainability. The results indicate that the proposed methods surpass other state-of-the-art approaches.\n- The paper acknowledges the limitations and suggests future directions for GraphCare, indicating the authors' awareness of potential challenges and their consideration of future improvements.\n- The authors conduct comprehensive ablation studies on the backbone models, adding robustness to their findings.", "weaknesses": "- The complexity of the proposed framework presents challenges for implementation. The paper would benefit from a more comprehensive explanation of the implementation specifics of the framework.\n- While the authors discuss the interpretability of GraphCare in Section 4.3, the depth of analysis is confined to observations on entity connections and their impact on model performance. It would be beneficial if the framework could uncover deeper insights in line with existing literature, or identify longer-distance relationships that might inspire new scientific discoveries.\n- In relation to the chosen experimental tasks, it would be helpful if the authors could elaborate on their selection criteria. Is there a particular preference for the information that the constructed graph helps in elucidating?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tNeoiRp9r6", "reviewer": "Reviewer_baD4", "summary": "The paper proposes a framework that leverages external medical knowledge generated by LLM to create patient-specific knowledge graphs (KGs) for improving healthcare predictions. In experiments on MIMIC-III and MIMIC-IV datasets, GraphCare outperforms baselines in multiple tasks, including mortality, readmission, length of stay, and drug recommendation. These results highlight the potential of external KGs in healthcare predictions, particularly in scenarios with limited data.", "questions": "- What is κ-hop subgraph in Section 1?\n\n- How is distance threshold δ selected for node and edge clustering?\n\n- What is AGGREGATE in equation 1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- There are extensive experiments including different prediction tasks, ablation studies on different parts of the model, KG sizes, and data sizes. These experiments are helpful to understand the impact of the proposed method. \n\n- The paper introduces a novel approach, leveraging Language Models (LM) to create a Knowledge Graph (KG). This innovative idea simplifies the KG generation process and ensures that the extracted knowledge is semantically rich and interpretable.\n\n- The knowledge graph provide interpretability of the connections among different EHR nodes.", "weaknesses": "- The performance of mortality prediction is questionable. While previous literature has reported AUPRC values exceeding 20 on the MIMIC dataset, the results presented in Table 2 fall notably short of this benchmark [1]. What leads to such a gap?\n\n- The paper delves into various engineering details, such as Knowledge Graph (KG) construction and attention initialization, without offering sufficient clarity. The notation in sec 3.3 are too complicated to follow. Many of them are randomly presented without definition. \n\n- The paper only compares with the baseline methods without leveraging external information / knowledge. However, it will be more important to compare with knowledge graph generated by other methods [2].\n \n\n[1] Purushotham, S., Meng, C., Che, Z., & Liu, Y. (2017). Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets.\n[2] Rotmensch, M., Halpern, Y., Tlimat, A. et al. Learning a Health Knowledge Graph from Electronic Medical Records. Sci Rep 7, 5994 (2017).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tAgPUUxwfc", "reviewer": "Reviewer_qgPW", "summary": "This paper focuses on clinical event prediction such as mortality and readmission prediction and drug recommendation using EHR datasets. The authors propose to build knowledge graphs for medical concepts in EHR datasets. They first extract relations for every medical concept and then apply node/edge clustering based on word embeddings. Finally, they design a node-level and visit-level attention method to predict medical events. Experimental results show that the proposed method can largely enhance the event prediction performance.", "questions": "1) In node/edge clustering, a node can be a phrase. What is the word embedding of such nodes? \n\n2) Above Equation (2), the authors said that reducing the size of node/edge embedding can handle the sparsity problem. What is the sparsity problem here, and why can the node/edge embedding have such a sparsity problem, given that they are dense vectors? If there indeed is a sparsity problem, can we decrease the initial embedding size?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper is well-written and easy to follow. \n\n2) The proposed knowledge graph for medical concept built by LLM is interesting and novel. \n\n3) This paper conducts extensive experiments including ablation study and interpreta0bility analysis.", "weaknesses": "1) The discussion on related work could be more comprehensive. For instance, there are several new developments on the below aspects: \n\n* Recent work about knowledge graph extraction using LLMs. \n\n[1] Yao, Liang, Chengsheng Mao, and Yuan Luo. \"KG-BERT: BERT for knowledge graph completion.\" arXiv preprint arXiv:1909.03193 (2019). \n\n[2] Wang, Chenguang, Xiao Liu, and Dawn Song. \"Language models are open knowledge graphs.\" arXiv preprint arXiv:2010.11967 (2020). \n\n[3] Chen, Chen, et al. \"Knowledge Is Flat: A Seq2Seq Generative Framework for Various Knowledge Graph Completion.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022. \n\n[4] Lovelace, Justin, and Carolyn Rose. \"A Framework for Adapting Pre-Trained Language Models to Knowledge Graph Completion.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. \n\n[5] Chen, Chen, et al. \"Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting.\" Findings of the Association for Computational Linguistics: ACL 2023. 2023. \n\n* Regarding the bi-attention method, the below paper may be related:  \n\n[6] Lu, Chang, Chandan K. Reddy, and Yue Ning. \"Self-supervised graph learning with hyperbolic embedding for temporal health event prediction.\" IEEE Transactions on Cybernetics (2021). \n\n* The latest baseline, StageNet, in the evaluation tasks is published in 2020. New papers have been published recently: \n\n[6] Yang, Kai, et al. \"KerPrint: local-global knowledge graph enhanced diagnosis prediction for retrospective and prospective interpretations.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 4. 2023. \n\n[7] Lu, Chang, et al. \"Context-aware health event prediction via transition functions on dynamic disease graphs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 4. 2022. \n\n[8] Ma, Xinyu, et al. \"Patient Health Representation Learning via Correlational Sparse Prior of Medical Features.\" IEEE Transactions on Knowledge and Data Engineering (2022). \n\n[9] Xu, Yongxin, et al. \"SeqCare: Sequential Training with External Medical Knowledge Graph for Diagnosis Prediction in Healthcare Data.\" Proceedings of the ACM Web Conference 2023. 2023. \n\n[10] Sun, Ziyou, et al. \"EHR2HG: Modeling of EHRs Data Based on Hypergraphs for Disease Prediction.\" 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2022. \n\nAlthough these baselines may not be used in all tasks 1, 2, and 3, the base model can be interchangeable. The authors should pick at least a few of them for comparison. \n\n2) The knowledge graph building is somewhat unclear. A knowledge graph usually contains edges with multiple types. However, the edge type is not reflected in KG extraction, edge clustering, and the graph neural network. \n\n3) One suggestion is to compare more clustering methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)