RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records (2024)

Abstract: We present RAM-EHR, a Retrieval AugMentation pipeline to improve clinical predictions on Electronic Health Records (EHRs). RAMEHR first collects multiple knowledge sources, converts them into text format, and uses dense retrieval to obtain information related to medical concepts. This strategy addresses the difficulties associated with complex names for the concepts. RAM-EHR then augments the local EHR predictive model co-trained with consistency regularization to capture complementary information from patient visits and summarized knowledge. Experiments on two EHR datasets show the efficacy of RAM-EHR over previous knowledge-enhanced baselines (3.4% gain in AUROC and 7.2% gain in AUPR), emphasizing the effectiveness of the summarized knowledge from RAM-EHR for clinical prediction tasks. The code will be published at https://github.com/ritaranx/RAM-EHR.

heading: 1 Introduction
text: Electronic Health Records (EHRs), encompassing detailed information about patients such as symptoms, diagnosis, and medication, are widely used by physicians to deliver patient care. Recently, a vast amount of deep learning models have been developed on EHR data (Choi et al., 2020; Gao et al., 2020; Wang et al., 2023a) for various downstream prediction tasks (e.g., disease diagnosis, risk prediction) to facilitate precision healthcare.
To further improve the downstream predictive performance, several works attempt to augment the EHR visits with external knowledge. For example, van Aken et al. (2021) and Naik et al. (2022) incorporate additional clinical notes, although these clinical notes can be noisy and contain irrelevant contents for clinical predictions; another solution is to leverage external clinical knowledge graphs (KGs), such as UMLS (Chandak et al., 2023),
* Equal contribution.
which contain rich medical concepts (e.g., disease, medications) and their corresponding relationships. Integrating KGs with EHRs has been shown to boost model performance (Xu et al., 2023b; Gao et al., 2023). However, these works mostly rely on knowledge from a single source and medical KGs mainly focus on specific types of relations (e.g., hierarchical relations), which do not comprehensively capture the semantic information for medical codes (e.g., phenotype). Besides, it is nontrivial to align medical codes in EHRs with KGs due to the non-uniformity of surface names (e.g., abbreviations or colloquial terms) (Hao et al., 2021; Zhang et al., 2022). There also exist methods that use knowledge generated from large language models (LLMs) to assist EHR prediction (Jiang et al., 2024), but LLMs may not always provide the most relevant knowledge for target tasks and face the risk of hallucination. Effectively leveraging external knowledge to facilitate EHR predictive tasks remains a significant challenge.
In this work, we propose RAM-EHR, a retrievalaugmented framework tailored for clinical predictive tasks on EHRs. Instead of leveraging a single knowledge source, RAM-EHR collects multiple knowledge sources (e.g., KGs, scientific literature) and converts them to text corpus, which enjoys the merits of a more comprehensive coverage of knowledge in a unified format. Then, to obtain unified representations for different knowledge sources, we leverage dense retrieval (DR) (Karpukhin et al., 2020; Lin et al., 2023) to encode corpus and medical codes as dense vectors, intuitively capturing the semantics of medical codes and addressing the alignment issue between EHR and external knowledge. Finally, to reduce irrelevant information, we utilize an LLM to summarize the top-retrieved passages into concise and informative knowledge summaries relevant to downstream tasks for each medical code. This process enhances the relevance and utility of the retrieved knowledge for clinical tasks.
754
To leverage external knowledge to assist clinical prediction, we introduce a retrieval-augmented model alongside the local EHR predictive model, which relies solely on patient visit information. The augmented model concatenates summarized passages and medical codes, feeding them into a moderate-size, pre-trained language model. We then co-train the local model and the augmented model with a consistency regularization, which captures the complementary information from patient visits and summarized knowledge and helps the model with better generalization (Wan, 2009).
We verify the effectiveness of RAM-EHR by conducting experiments on two EHR datasets and show that RAM-EHR outperforms strong knowledge-enhanced predictive baselines by 3.4% in AUROC and 7.2% in AUPR on average. Our analysis further confirms the advantage of leveraging multi-source external knowledge as well as retrieval augmentation as plugins to assist vanilla EHR predictive models based on visits only. Additional studies justify the usefulness of summarized knowledge for assisting clinical prediction tasks.

heading: 2 Methodology

heading: 2.1 Problem Setup
text: The EHR data consists of a group of patients P with corresponding hospital visits V = {v1, v2, ..., v|V |}. Each visit vi includes a set of medical codes Ci ⊂ C, where C is the total set of medical codes for P . In this study, C contains multiple types of medical codes including diseases, medications, and procedures. Each medical code ci ∈ Ci is a clinical concept, and it is associated with a name si in the form of short text snippets. Given the clinical record vi with the involved medical codes Ci, we aim to predict the patient’s clinical outcome yi (a binary label).
Figure 1 presents a comprehensive workflow of RAM-EHR, with a specific focus on dense retrieval from multiple knowledge sources and consistency regularization with co-training.

heading: 2.2 Retrieval Augmentation w/ Medical Codes
text: Existing approaches often treat each visit as context-free vectors, which fail to capture the concrete semantics of medical codes. Being aware of this, we aim to create the summarized knowledge for each medical code ci using its surface name si via retrieval augmentation with additional contexts. Multi-source Corpus Creation. Retrieval augmentation requires additional corpora as external
knowledge. To ensure the coverage of clinical knowledge, we collect a diverse external resources M = {d1, d2, . . . , d|M|}. We represent each knowledge unit as a raw text to facilitate retrieval. The detailed information of M is in Appendix C.
Passage Retrieval. Given a collection of |M| passages, the objective of the retriever is to transform passages in a dense vector, so that it can efficiently retrieve the most relevant information to the input query. In our work, we adopt Dragon (Lin et al., 2023), a dual-encoder model with strong performance across domains as the retriever. Specifically, we first use the passage encoder RD(·) to build an index for corpus M to support retrieval. Then, at runtime, we use the query encoder RQ(·) to map the input to an embedding (same dimension as the passage embedding) and calculate the similarity as f(q, d) = RQ(q)
⊤RD(d). For the medical code ci with the surface name si, we retrieve top-k (k = 5 in this work) passages Ti from the corpus M as
Ti = Top- k d∈M f(si, d). (1)
The top retrieved passages are considered as the external knowledge for the medical code ci.
Summarized Knowledge Generation. Although Ti contains the most relevant information for ci from M, directly using them to assist predictions can be suboptimal, as simply concatenating these passages often leads to long contexts, and some of the retrieved passages can also be irrelevant (Yu et al., 2023). Motivated by the fact that LLMs have strong capabilities in text summarization (Zhang et al., 2024), we propose to use the off-the-shelf LLM (gpt-3.5-turbo-0613) to generate the summarized knowledge ei for medical code ci as ei = LLM([Prompt, ti,1, · · · , ti,k]), (2) where ti ∈ Ti stands for the retrieved passages in Eq.(1). We incorporate information related to the downstream task within our prompt to ensure the generated summaries are task-specific. Detailed prompt designs can be found in Appendix F.
Remark. The retrieval step is efficient as the corpus indexing only needs to be done once before applying to prediction tasks. It only needs one extra ANN retrieval operation per query, which is efficiently supported by FAISS (Johnson et al., 2021). Besides, we cache the summarized knowledge for each medical code to avoid redundant operations.

heading: 2.3 Augmenting Patient Visits with Summarized Knowledge via Co-training
text: Recall that patient visits and summarized knowledge encode complementary information for clinical prediction tasks — visits capture cooccurence relationships, while summarized knowledge encodes semantic information. To effectively aggregate these two types of information, we design a co-training approach, detailed as follows. Augmented Model gϕ with Summarized Knowledge. For patient pi having the hospital visit vi with involved medical codes Ci, we decompose Ci into three subsets: Cdi for diseases, Cmi for medications, and Cpi for procedures. For each type of medical code, we flatten the visit into a document by concatenating all the codes and their summarized knowledge in a reversed sequential order. For example, for disease code Cdi , the flattened document can be Xdi = {[CLS], Dt, Dt−1, . . . , D1}, where Di = ||c∈Di(c, e) is the concatenation of disease code and its summarized knowledge (Eq. 2) within the i-th visit. We then use a pre-trained language model (PLM) with a multi-layer perceptron (MLP) classification head as gϕ for prediction with flattened documents as inputs:
hki = PLM(X k i ), ŷi,1 = MLP ( ||k∈Shki ) . (3)
Here S = {p,m, d}, hi is the representation of [CLS] token of Xi, ŷi,1 is the prediction for the target task. We share PLM weights for three types of medical codes to improve efficiency. Local Model fθ with Visit Information. To harness the visit-level information, various deep learning architectures have been proposed. In principle, gϕ can be combined with any fθ to improve
performance. In main experiments, we use a hypergraph transformer (HyGT, Cai et al. (2022); Xu et al. (2023a)) due to its strong ability to capture high-order relationships between visits and medical codes. It first builds hypergraphs G = (V, E) by treating medical codes as nodes and patients as hyperedges, then leverages self-attention for aggregating neighborhood information. The details for HyGT are in Appendix E. We obtain the prediction ŷi,2 with fθ as
ei = HyGT(G, Vi), ŷi,2 = MLP(ei), (4) where ei is the representation of patient i after hypergraph transformer. Co-training. We integrate the two predictors into a co-training framework, with the learning objective:
Laug =E(Vi,yi)∼P ℓ(ŷi,1, yi) + λDKL(ŷi,1, ỹ), Lloc =E(Vi,yi)∼P ℓ(ŷi,2, yi) + λDKL(ŷi,2, ỹ),
(5) where ℓ(·) is the binary cross-entropy loss, ỹ = βŷi,1 + (1− β)ŷi,2, λ, β are two hyperparameters. Two losses in Eq. 5 are designed to encourage fθ and gϕ regularize each other, which can stabilize the learning for two models. During the inference stage, we directly use the ỹj as the final prediction for the j-th test example pj .

heading: 3 Experiments

heading: 3.1 Experiment Setups
text: ⋄ Datasets. We conduct experiments on the public MIMIC-III dataset (Johnson et al., 2016) and a private CRADLE dataset collected from a large healthcare system in the United States. We perform a 25-label phenotypes prediction task on MIMICIII, and a cardiovascular disease (CVD) endpoints
prediction task for diabetes patients on CRADLE. We randomly split them into train/validation/test sets by 7:1:2. We present the detailed statistics of MIMIC-III and CRADLE in Table 1. Please refer to Appendix B for details. ⋄ Evaluation Metrics. Following Choi et al. (2020), we employ Accuracy, AUROC, AUPR, and Macro-F1 as evaluation metrics, where AUROC is the main metric. For accuracy and F1 score, we use a threshold of 0.5 after obtaining predicted results. ⋄ Baselines. We consider three groups of baselines: (a) Predictive models with visit information only: (1) Transformers (Li et al., 2020); (2) GCT (Choi et al., 2020); (3) HyGT (Cai et al., 2022); (b) Predictive models with external knowledge: (4) MedRetriever (Ye et al., 2021); (5) GraphCare (Jiang et al., 2024); (c) Predictive models with clinical notes: (6) CORE (van Aken et al., 2021); (7) BEEP (Naik et al., 2022). See Appendix D for more details. ⋄ Implementation Details. In this work, we use Dragon (Lin et al., 2023) as the dense retriever, with the passage encoder RD(·)1 and the query encoder RQ(·)2. We use k = 5 during the retrieval stage without tuning. We choose UMLSBERT (Michalopoulos et al., 2021) for RAM-EHR and relevant baselines as gϕ, with a maximum length of 512, and HyGT (Cai et al., 2022) as fθ in main experiments, but RAM-EHR can be adapted to multiple gϕ and fθ (Sec 3.3). We set the learning rate to 5e-5 for gϕ and 1e-4 for fθ, batch size to 32, and the number of epochs to 5. We select β, λ based on the performance of the validation set, and present the parameter study in Appendix G. For the model training, all the experiments are conducted on a Linux server with one NVIDIA A100 GPU.

heading: 3.2 Main Experimental Results
text: Table 2 exhibits the experiment results of RAMEHR and baselines. First, we observe RAM-EHR surpasses baselines lacking external knowledge,
1https://huggingface.co/facebook/ dragon-plus-query-encoder
2https://huggingface.co/facebook/ dragon-plus-context-encoder
highlighting the benefits of retrieval augmentation. Second, RAM-EHR outperforms knowledgeenhanced baselines due to the diverse collection of external knowledge as well as the co-training scheme that leverages information from both visit and semantic perspectives. Third, directly using medical notes leads to inferior outcomes due to potential irrelevance, whereas combining medical codes with summarized knowledge as RAM-EHR proves more effective for prediction tasks.

heading: 3.3 Additional Studies
text: Ablation Study. On the bottom of Table 2, we inspect different components in RAM-EHR and observe that removing any of them hurts the performance, which justifies the necessity of our designs. Besides, we observe that using the summarized knowledge with gϕ already achieves strong performance, highlighting the benefit of capturing the semantics of medical codes. Effect of fθ and gϕ. With various fθ and gϕ, we demonstrate the flexibility of RAM-EHR in Figure 2(a) and 2(b) by the consistent performance gain across different models. Notably, even with a lightweight fθ (Clin-MobileBERT) having only 25M parameters, RAM-EHR reaches close performance to UMLS-BERT, providing an efficient option for EHR predictive modeling. Effect of Information Source M. We then evaluate the effectiveness of each knowledge source within M. Figure 2(c) indicates that incorporating all corpus yields the highest performance, highlighting the value of diverse corpora. Besides, using Drugbank alone contributes minimally, likely due to its limited scope of medication information. Moreover, we observe that leveraging knowledge bases (e.g., MeSH) is more beneficial than literature sources, as they offer broader and more generic information conducive to clinical prediction tasks. Parameter Study. In Figure 3, we conduct parameter studies on both datasets for β and λ in Eq. 5. Figure 3(a) demonstrates that the model achieves the best performance when β is set to 0.2 and 0.4 on MIMIC-III and CRADLE, respectively, while the gain diminishes at the extremes. This highlights the contribution of combining the predictions from both the augmented model and the local model on the performance gain. In addition, λ is set to 1 and 5 on MIMIC-III and CRADLE, respectively, according to Figure 3(b). The positive values of λ indicate that the consistency loss enhances model performance.

heading: 3.4 Case Study
text: Figure 4 presents a case study on CRADLE to compare knowledge summarized by RAM-EHR and directly generated by LLM prompting. We observe that RAM-EHR provides more relevant information for the downstream task, particularly regarding the CVD outcome in this case, compared to direct LLM prompting. This also aligns with the human study evaluating the quality of 40 randomly sampled knowledge per type of code on a scale of [0,1,2] in Figure 2(d). The study on hyperparameters and retrieval components is in Appendix G.

heading: 4 Conclusion
text: We propose RAM-EHR, which uses dense retrieval with multiple knowledge sources and consistency regularization to enhance EHR prediction tasks. Experiments on two EHR datasets show the efficacy of RAM-EHR over baselines with a gain of
3.4% in AUROC and 7.2% in AUPR. In addition, we conduct human studies to confirm the utility of generated knowledge.

heading: Acknowledgement
text: We thank the anonymous reviewers and area chairs for valuable feedbacks. This research was partially supported by the Emory Global Diabetes Center of the Woodruff Sciences Center, Emory University. Research reported in this publication was supported by the National Institute Of Diabetes And Digestive And Kidney Diseases of the National Institutes of Health under Award Number K25DK135913. The research also receives partial support by the National Science Foundation under Award Number IIS-2145411. The content is solely the responsibility of the authors and does not necessarily rep-
resent the official views of the National Institutes of Health. We also thank Microsoft for providing research credits under the Accelerating Foundation Models Research Program.
Limitations
In this work, we propose RAM-EHR to unify external knowledge in text format and adapt it for EHR predictive tasks. Despite its strong performance, we have listed some limitations of RAM-EHR:
Building Multi-source Corpus M. In this study, we construct a multi-source corpus M by manually selecting five relevant sources within the clinical domain. In real-world scenarios, the grounding corpora usually require customization according to query domains and user needs. Therefore, effectively selecting grounding corpora and efficiently evaluating their relative contributions remains an unresolved issue. Furthermore, retrieved evidence may contain noise that could potentially degrade model performance, highlighting the importance of developing fine-grained filtering or re-ranking modules as a crucial area for future research.
Efficiency. The integration of the augmented model gϕ can result in additional time complexity. In our main experiment setups (using UMLSBERT), co-training usually takes 1.5× to 2× more times than using the local model alone. One potential solution is to use a lightweight model (e.g., Clin-MobileBERT) to improve efficiency.
Ethical Considerations
One potential ethical consideration concerns the use of credential data (MIMIC-III and CRADLE) with GPT-based online services. We have signed and strictly adhered to the PhysioNet Credentialed Data Use Agreement3 for the legal usage of the MIMIC-III dataset. To prevent sensitive information from being shared with third parties through APIs, we carefully follow the guidelines4 for the responsible use of MIMIC data in online services. Specifically, we have requested to opt out of human review of the data by filling out the Azure OpenAI Additional Use Case Form5 in order to utilize the Azure Open AI service while ensuring that Microsoft does not have access to the patient
3https://physionet.org/about/licenses/ physionet-credentialed-health-data-license-150/
4https://physionet.org/news/post/ gpt-responsible-use
5https://aka.ms/oai/additionalusecase
data. The utilization of LLMs in our framework is strictly for the purpose of building medical conceptspecific KGs. In addition, the building of medical concept-specific KGs does not involve direct interaction with any individual patient information. We iterate through all concepts in the medical coding system (e.g., CCS and ICD) to generate their respective KGs using LLMs, and these KGs are stored locally.

heading: A Related Works
text: Retrieval Augmented Learning and its Application in Clinical Domain. Retrieval augmented learning, which collects additional contextual information from external corpus, has shown effectiveness on diverse tasks including language modeling (Borgeaud et al., 2022), knowledge-intensive NLP (Lewis et al., 2020; Shi et al., 2023), commonsense reasoning (Wang et al., 2021), code genereation (Parvez et al., 2021), and few/zeroshot learning (Izacard et al., 2023). Compared to the general domain, the application of retrieval augmented learning to clinical tasks is still underexplored. Some efforts have been paid to retrieval augmented clinical language models (Zakka et al., 2024) as well as clinical question answering (Wang et al., 2023b; Jeong et al., 2024). The most relevant works are Ye et al. (2021); Naik et al. (2022), which leverage clinical literature to augment clinical predictive models. Compared to these works, our contribution lies in two folds: (1) we design a retrieval augmentation for structured EHRs with a diverse collection of external knowledge, which provides more relevant information for target clinical prediction tasks; (2) we incorporate a co-training scheme to leverage both the visit-level information and external knowledge for predictions.
Knowledge-enhanced EHR Predictive Models. Many studies attempt to harness external knowledge for clinical prediction tasks. The majority of them leverage structured knowledge, such as medical ontology (Choi et al., 2017; Panigutti et al., 2020), to capture hierarchical relationships among medical codes, or employ personalized knowledge graphs (Xu et al., 2023b; Jiang et al., 2024) to integrate patient-specific information. However, these methods often suffer from limited coverage of all medical codes due to the complexity of surface names. Alternatively, some approaches utilize unstructured medical text for health prediction tasks (Ye et al., 2021). However, Ye et al. (2021) rely on a restricted corpus of approximately 30,000 passages as their external corpus, resulting in limited coverage.

heading: B Task Information
text: MIMIC-III. The MIMIC-III dataset (Johnson et al., 2016) is a large, freely available database that contains de-identified health-related data from over 4,000 patients who stayed in critical care units at
Algorithm 1 Overview of RAM-EHR. 1: Input: P: patients; V : corresponding hospital visits of
patients. 2: Initializing multi-source external knowledge M; 3: for i = 1, · · · , |V | do 4: for ci ∈ vi do 5: Get the medical code ci and the corresponding textual name si included in visit vi; 6: // Passage Retrieval 7: Retrieve passages Ti via Eq. (1) 8: // Knowledge Summarization (Accelerated with caching) 9: Summarize knowledge ei for ci via Eq. (2); 10: end for 11: // Co-training 12: Predict ŷi,1 with knowledge-augmented model gϕ via Eq. (3); 13: Predict ŷi,2 with visit-based local model fθ via Eq. (4); 14: // Update Model Parameters 15: Compute loss function L via Eq. (5); 16: Update model parameters ϕ and θ; 17: end for
Output: Augmented model gϕ and local model fθ; Final prediction ỹj = βŷj,1 + (1 − β)ŷj,2 for the j-th test example pj .
the Beth Israel Deaconess Medical Center between 2001 and 2012. We conduct the phenotyping prediction task proposed by (Harutyunyan et al., 2019). It aims to predict whether the 25 pre-defined acute care conditions (see Table 3) are present in a patient’s next visit, based on the information from their current visit. The problem is formulated as a 25-label binary classification, considering that multiple phenotypes may exist in a single visit. For data preprocessing, we focus on patients with multiple hospital visits, identified based on their admission information. We extract pairs of consecutive visits for each patient. For each pair, we extract diseases, medications, and procedures from the health records in the former visit as input, and identify the phenotypes in the latter visit as labels, using Clinical Classifications Software (CCS) from the Healthcare Cost and Utilization Project (HCUP)6.
CRADLE. For the CRADLE dataset, we conduct a CVD outcome prediction task, which predicts whether patients with type 2 diabetes will experience CVD complications within 1 year after their initial diagnosis, including coronary heart disease (CHD), congestive heart failure (CHF), myocardial infarction (MI), or stroke. Diseases are identified by their ICD-9 or ICD-10 clinical codes. The usage of data has been approved by the Institutional Review Board (IRB).
6https://hcup-us.ahrq.gov/toolssoftware/ccs/ AppendixASingleDX.txt

heading: C Knowledge Sources
text: C.1 Descriptions • PubMed7: PubMed is a free search engine
accessing primarily the MEDLINE database of references and abstracts on life sciences and biomedical topics. It provides users with access to millions of scientific documents, including research papers, reviews, and other scholarly articles. We use the Entrez package to extract the PubMed articles8, resulting in 230k documents.
• DrugBank9 (Wishart et al., 2008): DrugBank is a comprehensive and freely accessible online database containing information on drugs and drug targets. It integrates detailed drug data (chemical, pharmacological, and pharmaceutical) with comprehensive information on drug targets (sequence, structure, and pathway). We use the data from the original database, which contains 355k documents.
• Medical Subject Headings (MeSH)10: Medical Subject Headings (MeSH) is a comprehen-
7https://pubmed.ncbi.nlm.nih.gov/ 8https://biopython.org/docs/1.75/api/Bio.
Entrez.html 9https://go.drugbank.com/releases/latest
10https://www.ncbi.nlm.nih.gov/mesh/
sive controlled collection for indexing journal articles and books in the life sciences. It organizes information on biomedical and health-related topics into a hierarchical structure. The corpus contains 32.5k documents covering various medical concepts.
• Wikipedia11 (Vrandečić and Krötzsch, 2014): Wikipedia is a free, web-based, collaborative, multilingual encyclopedia project that is supported by the non-profit Wikimedia Foundation. We extract web pages that contain medical-related information by using the medical codes list (e.g., ICD10 and ATC), resulting in 150k documents.
• KG12 (Chandak et al., 2023): We use PrimeKG in our experiments. It offers a comprehensive overview of diseases, medications, side effects, and proteins by merging 20 biomedical sources to detail 17,080 diseases across ten biological levels. For this study, we select knowledge triplets that contain medical codes within three types (disease, medication, procedure) used in this work, resulting in 707k triplets. We use the template in Appendix C.2 to transform these triplets into sentences.
C.2 Translating Format
We list the template to transform knowledge triplets into sentences in KG as follows:
candidate_relation = ["disease_phenotype_positive", "disease_protein", "disease_disease", "drug_effect", "drug_protein"]
relations = { "phenotype present": "[ent1] has the phenotype [ent2]", "carrier": "[ent1] interacts with the carrier [ent2]", "enzyme": "[ent1] interacts with the enzyme [ent2]", "target": "The target of [ent1] is [ent2]", "transporter": "[ent2] transports [ent1]", "associated with": "[ent2] is associated with [ent1]", "parent-child": "[ent2] is a subclass of [ent1]", "side effect": "[ent1] has the side effect of [ent2]" }
11https://www.wikipedia.org/ 12https://github.com/mims-harvard/PrimeKG

heading: D Baseline Information
text: • Transformer (Li et al., 2020): It leverages the Transformer (Vaswani et al., 2017) architecture to model sequential EHR visits for clinical prediction tasks.
• GCT (Choi et al., 2020): It employs the Transformer model to learn the EHR’s hidden structure via medical codes. Additionally, it introduces the Graph Convolutional Transformer, integrating graph neural networks to utilize the EHR structure for prediction.
• HyGT (Cai et al., 2022): It leverages hypergraph transformers that regard patients as hyperedges and medical codes as nodes for EHR predictive tasks.
• MedRetriever (Ye et al., 2021): It retrieves the most relevant text segments from a local medical corpus using string similarity. Then, it uses query features aggregated with EHR embeddings and disease-specific documents via self-attention.
• GraphCare (Jiang et al., 2024): It generates personalized knowledge graphs via prompting LLMs and leverages attention-based graph neural networks for healthcare predictions.
• CORE (van Aken et al., 2021): It integrates clinical knowledge with specialized outcome pre-training, and uses language models to predict clinical notes for prediction.
• BEEP (Naik et al., 2022): It augments the language models with the retrieved PubMed articles and fuses them with information from notes to predict clinical outcomes.

heading: E Details for Hypergraph Transformer
text: First of all, we construct a hypergraph G = (V, E) based on EHR data, where each patient visit is represented as a hyperedge connecting to all medical codes associated with the visit as nodes. Then we utilize HyGT (Cai et al., 2022) to jointly learn the node and hyperedge embeddings. Specifically, The hyperedge embeddings aggregate information from nodes within each hyperedge, while the node embeddings aggregate information from hyperedges connecting the nodes. In the l-th neural network
layer, the node and hyperedge embeddings are updated as
X(l)v = fE→V ( Ev,E(l−1) ) , (6) E(l)e = fV→E ( Ve,X(l−1) ) , (7)
where X(l)v and E (l) e represent the embeddings of node v and hyperedge e in the l-th layer (1 ≤ l ≤ L), respectively. Ev,E denotes the hidden representations of hyperedges that connect the node v, while Ve,X is the hidden representations of nodes that are contained in the hyperedge e. The two message-passing functions fV→E(·) and fE→V(·) utilize multi-head self-attention (Vaswani et al., 2017) to identify significant neighbors during propagation as
fV→E(S) = fE→V(S) = Self-Att(S),
where S is the input embedding for the attention layer, Self-Att(S) = LayerNorm(Y + FFN(Y )). Y is the output from the multihead self-attention block Y = LayerNorm(S +∥∥h i=1
SAi(S)), SAi(S) denotes the scaled dotproduct attention:
SAi(S) = softmax
( WQi (SW K i )
⊤ √ ⌊d/h⌋
) SW Vi .
WQi , W K i , and W V i are learnable parameters for the i-th head corresponding to queries, keys, and values, respectively. To interpret the above process, the input sequence S is projected into different h heads. The output of each head is then concatenated (denoted by
∥∥) to form the multi-head attention output. This output of multi-head attention layer Y is then fed into a feed-forward neural network (FFN), comprising a two-layer Multilayer Perceptron (MLP) with ReLU activation functions.

heading: F Details for Prompt Design
text: We present the detailed design of the prompt template as follows:
Prompt for LLM Summarization
Suppose you are a physician working on a health -related outcome prediction task and need to get relevant information for the given <task>. Here is relevant information: <medical code type> Name: <medical code name> Retrieve Passage #1: <retrieved document 1> Retrieve Passage #2: <retrieved document 2>
... Based on the above information, Could you
generate 1 sentence of around 10-20 words to summarize the knowledge for the
<medical code type> that is useful for the <task>?
<task> is the brief description of the downstream task. <medical code type> is either “disease”, “medication” or “procedure”, depending on the input.

heading: G Additional Experiment Results
text: We evaluate the effect of the augmented model gϕ, the local model fθ, and the knowledge source M on CRADLE in Figure 5. The experimental results further demonstrate that both models and different knowledge sources contribute to the performance gain. Moreover, it is observed that RAM-EHR is flexible to be applied upon different models, with a comparable performance with RAM-EHR.
For the human studies in Section 3.3, we provide the following guidelines for the annotators to evaluate the quality of the generated knowledge.
The goal of this evaluation is to assess the helpfulness of generated knowledge explaining or relating to specific medical codes in the context of target prediction tasks. Helpfulness is defined by the relevance, accuracy, and utility of the information in facilitating understanding or decision-making related to medical coding and its implications for predictive tasks.
Please rate the following generated knowledge with score 0, 1 or 2.
> 0: Irrelevant Definition: The knowledge does not provide any relevant information related to the medical code in question. It might be factually accurate but completely off-topic or not applicable to the context of target prediction tasks.
> 1: Partially Relevant and Useful
Definition: The knowledge provides some relevant information but either lacks completeness, specificity, or direct applicability to target prediction tasks. It might include general facts or insights that are related to the medical code but does not fully support decision-making or understanding in a predictive context.
> 2: Very Useful Definition: The knowledge directly addresses the medical code with accurate, relevant, and comprehensive information that is highly applicable to target prediction tasks. It should provide detailed understanding, or specific examples that facilitate decision-making, understanding, or application in predictive modeling.

heading: H Cost Information
text: Utilizing GPT-3.5-turbo as our base LLM model for generating summarized knowledge, we observe an average retrieval augmentation cost of $0.0025 per medical code in MIMIC-III and $0.0032 in CRADLE. Consequently, RAM-EHR does not result in excessive monetary expenses.

heading: Future Work
text: Furthermore, retrieved evidence may contain noise that could potentially degrade model performance, highlighting the importance of developing fine-grained filtering or re-ranking modules as a crucial area for future research. Efficiency. The integration of the augmented model gϕ can result in additional time complexity. In our main experiment setups (using UMLSBERT), co-training usually takes 1. 5× to 2× more times than using the local model alone. One potential solution is to use a lightweight model (e. g. , Clin-MobileBERT) to improve efficiency. Ethical Considerations
One potential ethical consideration concerns the use of credential data (MIMIC-III and CRADLE) with GPT-based online services. We have signed and strictly adhered to the PhysioNet Credentialed Data Use Agreement3 for the legal usage of the MIMIC-III dataset. To prevent sensitive information from being shared with third parties through APIs, we carefully follow the guidelines4 for the responsible use of MIMIC data in online services. Specifically, we have requested to opt out of human review of the data by filling out the Azure OpenAI Additional Use Case Form5 in order to utilize the Azure Open AI service while ensuring that Microsoft does not have access to the patient
3https://physionet. org/about/licenses/ physionet-credentialed-health-data-license-150/
4https://physionet. org/news/post/ gpt-responsible-use
5https://aka. ms/oai/additionalusecase
data. The utilization of LLMs in our framework is strictly for the purpose of building medical conceptspecific KGs. In addition, the building of medical concept-specific KGs does not involve direct interaction with any individual patient information. We iterate through all concepts in the medical coding system (e. g. , CCS and ICD) to generate their respective KGs using LLMs, and these KGs are stored locally