Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notes (2024)

Abstract: The automatic construction of knowledge graphs (KGs) is an important research area in medicine, with far-reaching applications spanning drug discovery and clinical trial design. These applications hinge on the accurate identification of interactions among medical and biological entities. In this study, we propose an end-to-end machine learning solution based on large language models (LLMs) that utilize electronic medical record notes to construct KGs. The entities used in the KG construction process are diseases, factors, treatments, as well as manifestations that coexist with the patient while experiencing the disease. Given the critical need for high-quality performance in medical applications, we embark on a comprehensive assessment of 12 LLMs of various architectures, evaluating their performance and safety attributes. To gauge the quantitative efficacy of our approach by assessing both precision and recall, we manually annotate a dataset provided by the Macula and Retina Institute. We also assess the qualitative performance of LLMs, such as the ability to generate structured outputs or the tendency to hallucinate. The results illustrate that in contrast to encoder-only and encoder-decoder, decoder-only LLMs require further investigation. Additionally, we provide guided prompt design to utilize such LLMs. The application of the proposed methodology is demonstrated on age-related macular degeneration. Data and Code Availability The dataset utilized in this study is provided by the Macula and Retina Institute and is not accessible to the public. Institutional Review Board (IRB) This research does not require IRB approval.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 295–317 August 16, 2024. ©2024 Association for Computational Linguistics
295
Data and Code Availability The dataset utilized in this study is provided by the Macula and Retina Institute and is not accessible to the public.
Institutional Review Board (IRB) This research does not require IRB approval.

heading: 1 Introduction
text: There are several biomedical data corpora available that provide valuable knowledge, and one
such source is PubMed (Kilicoglu et al., 2012). PubMed is a search engine that accesses MEDLINE (Kilicoglu et al., 2012), which is a database of abstracts of medical publications and references. Moreover, the widespread adoption of electronic medical records (EMR) has brought various opportunities for medical knowledge discovery. Knowledge graphs (KG) are often used for knowledge discovery, because graph-based abstraction offers numerous benefits when compared with traditional representations. They have been applied to various areas of healthcare, including identifying protein functions (Santos et al., 2022), drug repurposing (Drancé et al., 2021), and detecting healthcare misinformation (Cui et al., 2020). Another application may be a clinical trial design (Skelly et al., 2012), during which identification of confounding variables is an important step. Confounding variables may mask an actual association, or, more commonly falsely demonstrate an apparent association between the treatment and outcome when no real association between them exists.
KGs are a powerful tool for organizing and representing knowledge in a graph structure, where nodes represent entities within a specific domain, while edges symbolize relationships between these entities. The type of relationships may vary depending on the domain, allowing for the use of directed or undirected graphs. For example, in (Nordon et al., 2019), they employed a directed graph to encode causal relationships between diseases. Other KGs may utilize both symmetric and asymmetric relationships. In our work, we specifically focus on using directed graphs to represent relationships between diseases and various factors, treatments, and manifestations that coexist with a patient while experiencing the disease (referred to as ’coexists_with’).
Recent advancements in large language models (LLM) offer an opportunity to think about their ability to learn valuable representations from the knowl-
edge encoded in medical corpora. Effectively analyzing textual data and KG construction requires extensive domain knowledge and is often a timeconsuming process for medical experts. To address this challenge, we propose an end-to-end method for automatically constructing knowledge graphs from electronic medical record (EMR) notes using LLMs, specifically through relation extraction.
Previous studies have suggested the utilization of specific LLMs for clinical relation extraction (Agrawal et al., 2022; Sushil et al., 2022). However, due to the inherent safety-critical nature of healthcare, we conducted a comprehensive analysis of the performance and safety attributes of LLMs with varying architectures. To evaluate and assess their potential for medical applications and to address potential safety concerns, we introduced a manually annotated, private dataset and benchmarked the performance of 12 distinct LLMs. We have not performed an analysis on publicly available EMR datasets, such as MIMIC-III (Johnson et al., 2016), because some of the models have used these datasets for training or fine-tuning. Our analysis revealed that in contrast with encoder-only and encoder-decoder models, decoder-only models need further guidance to output in a structured manner, which is required for relation extraction to construct the KG. We, therefore, introduced a guided prompt design that helped to utilize some of such LLMs for our task and analyzed issues that are making others unsuitable. This rigorous assessment forms a critical foundation for the safe and effective deployment of LLMs in the healthcare domain. Our work takes the form of the following contributions:
• We present a end-to-end method leveraging LLMs for the automatic construction of KGs from EMR notes
• We conduct an extensive and rigorous evaluation of the performance of 12 LLMs of various architectures specifically tailored for clinical relation extraction
• We provide guided prompt design to utilize decoder-only LLMs for relation extraction to construct KG between aforementioned medical entities

heading: 2 Related Work
text: One notable success in the construction of knowledge bases (KBs) from biomedical textual data is
SemRep (Rindflesch and Fiszman, 2003). SemRep is a rule-based system that combines syntax and semantics with biomedical domain knowledge contained in the Unified Medical Language System (UMLS) (Bodenreider, 2004) for semantic relation extraction. The range of predicates in SemRep is diverse, including molecular interactions, disease etiology, and static relations. Shalit et al. (Nordon et al., 2019) further improve the precision of SemRep by adding three additional filtration steps.
As one may observe, SemRep utilizes various levels of language modeling. It has been experimentally demonstrated that LLMs intrinsically learn these levels of language specification, without explicit programming (Søgaard, 2021). In (Sung et al., 2021), BERT-based models with probing are used to extract relations between biomedical entities. The authors observe that, although LLMs can extract biomedical knowledge, they are biased towards frequently occurring entities present in prompts. We do not argue about the bias of LLMs, but rather the complexity of extracting relations via probing. We propose providing larger context information than that which is solely present in the prompt.
(Rotmensch et al., 2017) utilizes both structured and unstructured data from EMR to construct knowledge graphs. The structured data includes ICD-9 (International Classification of Diseases) diagnosis codes, while the unstructured data comprises various notes written by physicians and nurses to track a patient’s course. On the other hand, (Chandak et al., 2023) employs 20 multimodal data resources to describe a disease with various relationships representing different biological scales. However, in this work, we solely concentrate on clinical notes for information extraction and KG construction.
(Trajanoska et al., 2023) makes connection between LLMs and semantic reasoning to automatically generate a KG on the topic of sustainability. It further populates it with concrete instances using news articles from the internet. It experiments with REBEL (Huguet Cabot and Navigli, 2021) and ChatGPT and shows that ChatGPT (OpenAI, 2023) is able to automatically create KGs from unstructured text, if reinforced with detailed instructions.
The paper on few-shot clinical extraction using LLMs (Agrawal et al., 2022) discusses the challenge of extracting important variables from clinical data and presents an approach that leverages large language models, specifically InstructGPT
(Ouyang et al., 2022), for zero-shot and few-shot information extraction from clinical text. The authors demonstrate the effectiveness of this approach in several NLP tasks that require structured outputs, such as span identification, token-level sequence classification, and relation extraction. To evaluate the performance of the system, the authors introduce new datasets based on a manual reannotation of the CASI dataset (Moon et al., 2014).
We argue that our setup is more complex as we do not consider clean, well-written, academic corpora such as PubMed (Kilicoglu et al., 2012) and CASI (Moon et al., 2014). The EMR corpus contains a significant amount of grammatical errors ("there is some heme OD .. ?"). Practitioners use abbreviations and notations ("RTO") not defined in the context, obfuscating the underlying information even further. Our study benchmarks different LLMs of varying architectures and training procedures on this challenging dataset.

heading: 3 Dataset
text: For this cohort study, data was obtained from the EMR of the Macula & Retina Institute, an independent health system in Glendale, California, USA. The dataset included approximately 10,000 patient records of individuals with retina-related eye diseases who had visited the institute between 2008 and 2023. The study focused on extracting knowledge from the clinical notes, which are records of observations, plans, and other activities related to patient care. These notes contain a patient’s medical history and reasoning and can be used to identify complex disease-related patterns such as potential treatments, causes, and symptoms. In total, the study analyzed 360,000 notes relating to 122 unique eye diseases.

heading: 3.1 Dataset preprocessing
text: Clinical notes often include repetitive segments following a standardized template used by medical practitioners, resulting in unnecessary computational overhead during the analysis. To address this issue, cosine similarity is computed between the embeddings of notes generated by Sentence T5 XXL (Ni et al., 2022). If the similarity score exceeds the threshold (referred to as threshold_preprocessing, detailed in Appendix F), priority is given to the note with a higher word count to retain more informative content. Additionally, notes containing fewer than 5 words are excluded
from further analysis.

heading: 4 Proposed method
text: Our proposed method constructs a KG of diseases and their factors, treatments, and manifestations that the patient exhibits while undergoing the disease. To achieve this, the system initially identifies disease-specific notes as described in Subsection 4.1. Next, for each category of medical entity, we design set of questions (Subsection 4.5). We leverage an LLM to answer a pre-designed set of questions, taking into consideration the aforementioned disease-specific notes as contexts as described in Subsections 4.3 and 4.6. The list of LLMs that we experimented with are available in Subsection 4.2. All the experiments are performed on 8xV100 (32GB VRAM) GPUs which are widely accessible nowdays. The Subsection 4.7 discuss postprocessing techniques utilized to get the final relations to construct the KG.

heading: 4.1 Disease-specific notes identification
text: In clinical records, a single disease, denoted as dinput, may have multiple textual representations. The set of such expressions is denoted as Dinput. These expressions may vary between clinics as well. To identify all instances of dinput in the records, we employ the Unified Medical Language System (UMLS) Metathesaurus (Bodenreider, 2004), a comprehensive repository of biomedical terminologies and ontologies containing over 3 million concepts and their corresponding aliases, such as diseases, drugs, and procedures. We first check if any of the expressions in di ∈ Dinput appear in the records within our dataset, and if so, we add the record to a list of disease-specific records for dinput. Sometimes, clinicians may make typographical errors when recording the condition in the notes. To account for this, we use the BioBERT NER model to extract a list of diseases, denoted as Dnote, present in the record. We then calculate the cosine similarity between each expression dnotei ∈ Dnote and di ∈ Dinput. If the similarity is above threshold (denoted threshold_notes_identification, more in Appendix F) for at least one dnotei , we add the record to the list of relevant notes for the disease di. Refer to Appendix C for more details on the algorithm.

heading: 4.2 Models
text: Table 1 shows all the models that we used in this paper. Our main objective revolves around exper-
imenting with various architectures of LLMs and analyzing their performance through a comprehensive evaluation that brings forward potential edge cases and safety attributes. To accomplish this, we conducted experiments using different LLM models categorized under three architectures: encoderonly, decoder-only, and encoder-decoder. Our next objective was to include as much diverse LLMs as possible encompassing variations in size as well as the number of pretraining tokens. For more detailed insights into each individual model, please refer to Appendix A.

heading: 4.3 Aligning LLMs for relation extraction
text: In this work, we assume only query access to a large language model (i.e., no gradients). The task is to identify relations by finding answers to specific queries. We explore two distinct approaches for aligning large language models to the task: open-book QA (Gholami and Noori, 2021) and in-context learning (Brown et al., 2020).
QA aims to find an answer to a given query. In open-book QA, a query comprises a question and a context. The system attempts to find an answer to the question from the context. It utilizes various variations of BERT (Devlin et al., 2019) language models, as described in Table 1. The model consists of two sets of dense layers with sigmoid activation in addition to the based BERT model. The first layer seeks the start of the answer sequences, while the second layer seeks the end of the answer
sequences. For decoder-only and encoder-decoder models, we employ in-context learning (Brown et al., 2020), providing the LLM with a prompt consisting of a list of input-output pairs that answer a given query using the context. In this study, we focused on zero-shot (Wei et al., 2022), few-shot (Brown et al., 2020), and instruction-based prompting (Ye et al., 2023).

heading: 4.4 Prompt design
text: We follow a systematic and task-agnostic process to construct prompts as outlined in (Jimenez Gutierrez et al., 2022). As depicted in the examples in Figure 1, this method identifies three key components of a prompt: overall task instructions, a sentence introduction, and a retrieval message. In the case of zero-shot and few-shot approaches, simply entity-related questions are appended to the input (Figure 1 left-top). Additionally, for the fewshot approach, we provide an example input/output. For instruction-based prompting (Figure 1 leftbottom), overall task instructions are comprised of broad instructions for the task as it is described in (Jimenez Gutierrez et al., 2022).
Furthermore, we are introducing a prompt structure by defining a stringent input and response format. The primary focus is on extracting information exclusively from the provided context, accompanied by explicit instructions to incorporate specific entity types in the response. We have delineated a well-defined format for both the question and the response, promoting concise answers without explanations. Moreover, we have introduced a systematic approach to address situations where information is absent or questions are irrelevant, ensuring a consistent ’I do not know’ response. In essence, these modifications contribute to enhancing the clarity and precision of the model’s performance within this specific scientific context. For a comprehensive visualization of the refined prompt structure and its components, kindly consult the right block of Figure 1.

heading: 4.5 Question design
text: We define template questions like "What treats %s". The "%s" in the questions represents a placeholder for a disease. All the predicates (e.g. treats, affect, cause, factor) are taken from SemRep (Rindflesch and Fiszman, 2003). The questions are categorized into three types: treatment-related, factorrelated, and coexists_with-related questions. The treatment-related questions inquire about methods
to slow down the progression, decrease the chance, or reduce the risk of a specific condition. The factor-related questions aim to identify the causes, factors, or risks associated with a condition. The coexists_with-related questions explore any symptoms, effects, diseases, clinical tests, or behaviors that may manifest in the patient while experiencing the disease. The full list of questions for the LLM queries is available in Appendix B.

heading: 4.6 Relation extraction
text: We query an LLM for each disease d ∈ D with a question q(d) and a related context c ∈ C(d) (refer to Appendix C for more details). The LLM returns a list of answers with their corresponding probabilities for each query quartet ⟨d, q(d), c, t⟩ where t identifies the question type, i.e. treatment, factor, and coexists with. As a single probability estimate may be unreliable (Nordon et al., 2019), we keep the relation triplet ⟨e, t, d⟩ if the LLM has returned e as an answer to any question of category t more than relation_occurrence_number times and that the average probability over of the triplet is greater than relation_probability. For details on the choice of relation_occurrence_number and re-
lation_probability please refer to Appendix F. Finally, the category t with the highest probability is chosen as the final relation between e and d. Refer to Appendix C for more details.

heading: 4.7 Postprocessing
text: To map the model’s output to a list of values for each medical entity, we initially filtered out the predictions with a probability score lower than threshold (denoted prediction_probability, more in Appendix F). Subsequently, to remove meaningless information, stop words and punctuation were excised from each predicted text.
Furthermore, our approach involved addressing instances where the model conveyed uncertainty or lacked adequate context. When the large language model (LLM) produced responses such as "I do not know" due to ambiguity or insufficiency, we systematically filtered out these outputs.
Further analysis revealed that models tend to generate the same answers in various forms depending on the given context. For instance, predictions such as "areds" and "areds-2 vitamins" essentially refer to the same value for a specific medical entity, but are expressed differently. To address these varia-
tions, we employed normalized cosine similarity for the tokens in the model’s predictions. Specifically, for each medical entity, we calculated the cosine similarity between each pair of predictions. Predictions which exceed the similarity threshold (denoted similarity_postprocessing, more in Appendix F) were considered equivalent and subsequently grouped together. From each group, the prediction with the highest initial probability score assigned by the model was selected. Finally, the refined output was converted into a list of values, selecting spans of text directly from the LLM output. A qualitative example illustrating this process is provided in Appendix D.

heading: 5 Results
text: We now describe our experimental study over our techniques for constructing the KG.
Setup We construct a KG for age-related macular degeneration (AMD), a progressive eye disease predominantly affecting older individuals with a high incidence rate. Since KGs are typically too large to display directly, we provide their tabular representation instead. To reconstruct KGs from Tables 3 and 4, connect treatments listed in the Treatment column to AMD (the target entity) using arrows.
Similarly, connect factors from the Factor column to AMD, and establish connections to AMD with undirected edges for entities in the Coexists_with column.
The evaluation is based on precision and recall, which represent the ratio of correctly extracted terms by the model to all terms extracted by the model, and the ratio of correctly extracted terms by the model to all actual terms available in the clinical notes. The same metrics have been calculated for each entity (Treatment, Factor, and Coexists_with) separately. Therefore, the ground truth for comparison has been the entity values available in the clinical notes. Thus, we needed to review all clinical notes related to AMD and extract all factors, treatments, and ’coexists_with’ terms. You can find the explanation of these terms in Subsection 4.5. The AMD-related notes have been identified according to Subsection 4.1 and preprocessed as described in Subsection 3.1. These steps leave us with 320 clinical notes.We refer to the process of extracting terms as annotation. This annotation was carried out by two of the authors, a retina specialist, and a clinical research coordinator. To establish a consistent annotation schema, a set of examples was jointly annotated. Following this, each annotator independently annotated the same set of examples,
and the two sets of annotations were then combined via a joint manual adjudication process. As a result, we extracted 11 different treatments, 8 different factors, and 8 ’Coexists_with’ terms from the clinical notes.
Precision and recall results Table 2 shows the precision and recall results of different LLMs of various architectures. The best performance is consistently achieved with encoder-decoder LLMs for most medical entities. Specifically, FLAN-UL2, when used with our proposed prompt design, outperforms the other models. Furthermore, we observe that encoder-decoder models using 0-shot and few-shot prompting techniques are comparable to decoder-only models in some cases. However, when instruction-based or our proposed guided prompting is employed for encoder-decoder models, they significantly outperform the others.
Quantitative results for decoder-only models using 0-shot, few-shot, and instruction-based prompting techniques are not available. These models did not produce structured outputs, rendering them unsuitable for our task. Additional information can be found in Decoder-only models. Unlike other prompting techniques, guided instructionbased prompting (as described in Subsection 4.4) has demonstrated significant improvements. This prompt design allowed us to utilize only three decoder-only models for this task, out of the seven we experimented with. These models include Llama 2 (Touvron et al., 2023), Vicuna-33B (Zheng et al., 2023), and WizardLM-70B (Xu et al., 2023). The other four did not produce structured outputs with this prompt design, similar to the results obtained with the other three prompting techniques.
Notably, WizardLM-70B achieves the highest recall for factors and treatments, demonstrating that the incorporation of additional guidance has enhanced the understanding of the task by some of the decoder-only models, resulting in more precise and accurate answers. We believe that further research is required to explore the potential of decoder-only models for challenging relation extraction tasks, and future investigations may enhance their reliability. See prediction examples in Appendix E.
Decoder-only models Here we describe the challenges that make some of these models (BioGPT (Luo et al., 2022), OPT (Zhang et al., 2022), OPTIML-MAX (Iyer et al., 2022), Bloom (Scao et al., 2022)) with any of the prompting techniques were unsuitable for clinical relation extraction, thus KG
construction. Some of the models are prone to "hallucinating", a term commonly used to refer to the models generating responses that are factually incorrect or nonsensical. See such examples in Appendix E.2.1.
Furthermore, we observed cases where some models generated correct responses, but these responses did not originate from the given context. Another concern was the generation of excessively verbose or repetitive responses. Despite being contextually correct, the lengthy and redundant nature of these outputs complicated the postprocessing phase, making the integration of such responses into our KG construction pipeline impossible. See such examples in Appendix E.2.2.
Qualitative Example: AMD We continue using AMD as a qualitative example. AMD is a progressive eye disease affecting the retina, specifically the macula. The risk factors for AMD have been studied extensively and have widely been known to include age, race, smoking status, diet, and genetics (Holz et al., 2014; Heesterbeek et al., 2020). The exact reasons and mechanisms behind AMD are not yet fully researched. There are multiple pathways and factors for drusen formation and AMD progression, so it is hard to disentangle them. Large and numerous drusen are associated with an increased risk of developing advanced AMD (Schlanitz et al., 2019). The pathophysiologic landscape of AMD potentially involves degenerative transformations within several ocular components, including the outer retinal layers, the photoreceptors, retinal pigment epithelium (RPE) characterized by the loss of the ellipsoid zone (EZ) and atrophic changes, accumulation of subretinal/submacular fluids, perturbations in Bruch’s membrane leading to choroidal neovascularization (CNVM), and areas of choriocapillaris nonperfusion resulting in macular atrophy and fibrosis (Holz et al., 2014; Boyer et al., 2017). Medical evaluators annotated drusen, genetics, CNVM, smoking, RPE irregularities, submacular/subretinal fluid, fibrosis, and loss of EZ zone as risk factors for AMD. The KG constructed with the utilization of FLAN-UL2 with guided instruction-based prompting that have relatively the best quantitative performance, is visually presented in Table 3.
Notably, besides factors, the graph also highlights a spectrum of terms that are linked to potential treatments and symptoms associated with AMD. Among the treatment entities are ARED-
S/WACS vitamins, dietary interventions, and AntiVEGF treatments including Avastin and Lucentis. Other treatments indicated include PDT (Photodynamic Therapy), the utilization of Amsler grid, supplementation of Omega-3 fatty acids, and consumption of specific foods such as fish, spinach, and green leafy vegetables. The symptomatic aspects of AMD encompass a range of visual impairments and clinical manifestations. Patients afflicted with AMD often experience poor visual acuity, metamorphopsia (distorted vision), and can be diagnosed with either dry or wet AMD. Additionally, the management of the condition often involves undergoing assessments such as ForeseeHome and Macula Risk genetic testing, which play a pivotal role in monitoring the progression and development of AMD. Each of these terms is identified as values to the ’Coexists_with’ entity within the graph.
We also show the KG constructed by SemMedDB (Kilicoglu et al., 2012) in Table 4. SemMedDB is a repository of semantic predictions extracted from the titles and abstracts of all PubMed citations. It is evident that our approach has identified terms not found in the SemMedDB. Our method may not forge new terms where none existed in the original medical literature repository. However, the feedback from our medical evaluators underscores its potential
to contribute to novel discoveries by highlighting existing but overlooked information.

heading: 6 Conclusion
text: In this paper, we propose an end-to-end approach that harnesses LLMs for the automatic generation of KGs from EMR notes. KGs hold significant value in numerous healthcare domains, including drug discovery and clinical trial design. The entities involved in the KG construction process encompass diseases, factors, treatments, and manifestations that co-occur with patients experiencing these diseases. Through extensive evaluation across various LLM architectures, we have demonstrated that encoder-decoder models outperform others in clinical relation extraction. Additionally, we emphasize the need for additional investigation into the suitability of decoder-only models for medical applications, particularly given their critical safety implications. Furthermore, we provide guided prompt design to utilize these models. We believe that an automated knowledge extraction method may deliver substantial benefits to the medical community and facilitate further research in the field.

heading: 7 Limitations
text: The findings of the research are subject to several limitations. The primary one is that our experiments were conducted on a single dataset focused on one specific disease. This limitation arises from the necessity of annotations by medical practitioners, a process that is highly time-consuming. Furthermore, due to the private nature of our dataset, we opted to use only open-source models to ensure data privacy and security. While this approach safeguards patient information, it may limit the performance benefits that could be gained from proprietary models. Additionally, we assumed only query access to large language models (i.e., no gradients). Fine-tuning LLMs on a relevant corpus could potentially enhance their performance and accuracy, and this remains an area for future exploration.

heading: A Models
text: Encoder-only models Our approach utilizes a fine-tuned question-answering model based on BERT (Devlin et al., 2019), specifically fine-tuned on the SQuAD v2 dataset (Rajpurkar et al., 2016). This model, which we refer to as BERT-SQuAD-v2, benefits from the core principles of BERT, including random token masking during pretraining to encourage contextual understanding.
Inspired by advancements in the BERT family, we also incorporate RoBERTa (Liu et al., 2019), which is improved upon Bert by introducing a new pretraining recipe that includes training for longer and on larger batches, randomly masking tokens at each epoch instead of just once during preprocessing, and removing the next-sentence prediction objective. We also consider BioBERT (Lee et al., 2020), which is a pre-trained BERT model which is trained on different combinations of general & biomedical domain corpora.
Decoder-only models BioGPT (Luo et al., 2022), a generative Transformer model tailored for biomedical literature, has shown remarkable results on several biomedical NLP benchmarks, including an impressive 78.2% accuracy on PubMedQA (Jin et al., 2019). However, our efforts to employ BioGPT for relation extraction were met with challenges. The model frequently hallucinated during inference, making it unsuitable for our specific application in relation extraction.
Open Pretrained Transformers (OPT) (Zhang et al., 2022) represents a comprehensive suite of
decoder-only transformers designed for large-scale research. OPT-30B, a particular model from this suite, has been pre-trained predominantly on English text with some multilingual data from CommonCrawl. Sharing similarities with GPT-3, it uses a causal language modeling (CLM) objective. OPTIML (Iyer et al., 2022) represents an advanced version of the OPT model, enhanced with Instruction Meta-Learning. It’s been trained on an extensive collection known as the OPT-IML Bench, comprising roughly 2000 NLP tasks from 8 different benchmarks. Two variations exist: the standard OPT-IML trained on 1500 tasks, and OPT-IMLMax that covers all 2000 tasks.
BLOOM (Scao et al., 2022) stands as a sophisticated autoregressive Large Language Model (LLM), designed to produce coherent text across 46 languages and 13 programming languages, replicating human-like text generation capabilities.
Llama 2 (Touvron et al., 2023) is a distinguished collection of generative text models, with models ranging from 7 billion to 70 billion parameters. Presented by Meta, this repository encompasses the 70B variant, made compatible with the Hugging Face Transformers framework. Within the Llama 2 family lies a specialized series called Llama-2-70BChat, meticulously fine-tuned for dialogue-centric applications. This model excels, outstripping many open-source chat models in benchmarks and rivalling prominent closed-source counterparts like ChatGPT and PaLM in terms of helpfulness and safety.
Emerging from the wave of advanced chatbots, Vicuna-33B (Zheng et al., 2023) stands out as an open-source contribution, fine-tuned using the LLaMA framework based on dialogues from ShareGPT. Notably, when evaluated using GPT-4, Vicuna-33B not only showcased a commendable performance, rivaling the likes of OpenAI’s ChatGPT and Google Bard (achieving over 90%* quality), but also surpassed counterparts like LLaMA and Stanford Alpaca (Taori et al., 2023) in over 90%* of the tests. This exceptional achievement comes at a modest training cost of around $300, making Vicuna-33B an attractive proposition. Additionally, its code, weights, and a live demo are accessible for the research community, albeit restricted to non-commercial applications.
WizardLM-70B (Xu et al., 2023) is a Large Language Model (LLM) built on the foundation of LLaMA, incorporating a novel training approach known as Evol-Instruct. This method involves
leveraging artificial intelligence to evolve complex instruction data, setting WizardLM apart from LLaMA-based LLMs trained on simpler instructions. As a result it outperforms counterparts in tasks that demand intricate understanding and execution of instructions.
Encoder-decoder models FLAN-T5-XXL (Chung et al., 2022) is a encoder-decoder model that has been pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which each task is converted into a text-to-text format. It performs well on multiple tasks including question answering.
FLAN-UL2 (Yitayew, 2023) is an encoderdecoder model based on the T5 architecture. It uses the same configuration as the UL2 (Tay et al., 2022) model released earlier last year and was finetuned using the “Flan” prompt tuning and dataset collection (Wei et al., 2022). According to the original blog, there are some notable improvements over the original UL2 model. The Flan-UL2 checkpoint uses a receptive field of 2048 which makes it more usable for few-shot in-context learning. This Flan-UL2 checkpoint does not require mode tokens anymore.
In comparison to FLAN-T5, FLAN-UL2 outperforms FLAN-T5 XXL on all four setups with an overall decent performance lift of +3.2% relative improvement. Most of the gains seem to come from the CoT setup while performance on direct prompting (MMLU and BBH) seems to be modest at best.

heading: B Question list
text: C Algorithms
Algorithm 1 Disease-specific notes identification Ensure: result
result := {} Dinput = UMLS_Metathesaurus_API(dinput) for note in clinical_notes do
Dnote = BioBERT_NER(note) for di ∈ Dinput do
for dnotei ∈ Dnote do if note contains di then
result.append(note) else
similarity_score := calculate_cosine_similarity(dinput, dnotei) end if if similarity_score > threshold then
result.append(note) end if
end for end for
end for
Algorithm 2 Querying LLM Ensure: result
result := {} for d ∈ D do
for c ∈ C(d) do for t ∈ {treatment, factor, coexists_with} do
for q(d) ∈ Qt(d) do tmp := ⟨LM(⟨c, q(d)⟩), d, t⟩ ▷ Where LM returns a list of possible answers
▷ with their probabilities. result.insert(tmp)
end for end for
end for end for
Algorithm 3 Relation extraction Require: result from Algorithm 2
relation := {} Ensure: relations
for unique ⟨e, d, t⟩ in result do temp := ⟨average(result[e, d, t].score), count(result[e, d, t])⟩ if temp.average ≥ 0.1 and temp.count ≥ 10 then
stat← ⟨temp.average, e, d, t⟩ end if
end for for unique e, d in stat do
relations← ⟨d, argmaxt stat[e, d], e⟩ end for

heading: D Postprocessing

heading: E Prompts and sample outputs
text: E.1 Encoder-only models E.1.1 Examples of wrong predictions
Listing 1: BERT-SQuAD-v2: wrong prediction Question: What can slow the progression of
macular degeneration? Context: Macular Degeneration: Discussed the
nature of dry macular degeneration. Discussed Age Related Eye Disease Study and recommended AREDs vitamins for prevention purposes. Patient given Amsler grid to monitor for metamorphopsias or changes in central vision.
Answer: dry macular degeneration
Listing 2: RoBERTa-SQuAD-v2: wrong prediction Question: What does cause Macular Degeneration? Context: Macular Degeneration: Discussed the
nature of dry macular degeneration. Discussed Age Related Eye Disease Study and recommended AREDs vitamins for prevention purposes. Patient given Amsler grid to monitor for metamorphopsias or changes in central vision.
Answer: dry macular degeneration
E.1.2 Examples of correct predictions
Listing 3: BERT-SQuAD-v2: correct prediction Question: What can slow the progression of
myopic macular degeneration? Context: Myopic Macular Degeneration - Explained
that there is no specific treatment at this time. AREDS and WACS vitamins MAY help slow down the progression of the degeneration. Monitor closely. All questions were answered to the patient's satisfaction.
Answer: AREDS and WACS vitamins
Listing 4: BioBERT-SQuAD-v2: correct prediction Question: What does cause ARMD? Context: Macular Degeneration: Discussed the
nature of dry macular degeneration. Patient encouraged to use an Amsler grid to monitor macular function by looking for metamorphopsias or visual changes.
Answer: metamorphopsias or visual changes
Listing 5: RoBERTa-SQuAD-v2: correct prediction Question: What does cause amd? Context: Smoking is the largest `modifiable'
risk factor for AMD - by stopping smoking a patient can reduce the chance of getting AMD and of AMD progressing.There is a very strong association between smoking and macular disease. Smokers are up to four
times more likely than non-smokers to develop age-related macular degeneration ( AMD) and smokers with a genetic predisposition to AMD are eight times more likely to get the condition. Some research suggests people with certain genes can be 20 times more likely to get macular degeneration if they smoke.
Answer: Smoking
E.2 Decoder-only models E.2.1 Examples of hallucinations
Listing 6: BLOOM with instruction-based prompting: hallucination I want you to act as a question answering
machine. I will provide you with a questions and a context and you will reply with the answers. Question: What can slow the progression of armd? Instruction: If the answer is not in context
answer "i do not know". Context: Wet ARMD- Patient educated on condition.
Discussed diet and use of Macular Degeneration vitamins. All questions were answered to the patient's satisfaction.
Answer: i do not know
a: i think the question is a bit vague. it is not clear what the question is about. it is not clear what the context is. it is not clear what the answers are. it is not clear what the question is asking. it is not clear what the question is asking the answerer to do. it is not clear what the question is asking the answerer to do. it is not clear what the question is asking the answerer to do. it is not clear what the
Listing 7: BLOOM with guided instruction-based prompting: hallucination
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not
contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: coexists_with question: What does armd lead to? context: Acute Exudative ARMD OU - chronic
leakage OU. No significant changes OU. Recommend treating with Anti-VEGF injections . Discussed with patient the gravity of this disease, including the potential for vision loss and guarded recovery. Reviewed treatment options - Avastin, Lucentis, Eylea , PDT.
### Response effect:
If the question contains multiple entities, extract all of them, separated with commas. For example, if you have following question
Question 1:
Listing 8: BioGPT with guided instruction-based prompting: hallucination
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for
that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: factor question: What is the factor for armd? context: 2 small Druse OD- clinically does not
look like ARMD. Patient has a family history of ARMD, recommend starting on AREDS + WACS vitamins. Eat green leafy vegetables like Spinach 5 times a week and fish at least 2 times a week.
### Response I do not include any other entities to answer the
question.
Listing 9: OPT-30B with guided instruction-based prompting: hallucination
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: coexists_with question: What can amd convert to? context: Explained to patient that he does have
AMD but it is mild and not the cause of his blurred vision.
### Response effect: [ENTITY_1], [ENTITY_2], [ENTITY_3]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer Ï do not know.änd nothing more. No explanations. If the context does not contain the answer to the question, answer with the following format
### Response
I do not know. ### Input
question_type: factor
question: What can amd convert to?
context: Explained to patient that he does have AMD but it is mild and not the cause of his blurred vision. ### Response
factor: [ENTITY_1], [ENTITY_2], [ENTITY_3]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer Ï do not know.änd nothing more. No explanations. If the context does not contain the answer to the question, answer with the following format
### Response
I do not know. ### Input
question_type: treat
question: What can amd convert to?
context: Explained to patient that he does have AMD but it is mild and not the cause of his blurred vision. ### Response
treat: [ENTITY_1], [ENTITY_2], [ENTITY_3]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer Ï do not know.änd nothing more.
No explanations.
If the context does not contain the answer to the question, answer with the following format ### Response
I do not know. ### Input
question_type: factor
question: What can amd convert to?
context: Explained to patient that he does have AMD but it is mild and
E.2.2 Examples of wrong predictions
Listing 10: OPT-IML-MAX-30B with guided instruction-based prompting: wrong prediction
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: coexists_with question: What can armd convert to? context: Dry ARMD - Explained that there is no
specific treatment at this time. Can convert to wet ARMD, which is treatable. Eating a healthy diet that includes green leafy vegetables, fish, taking AREDS + WACS eye vitamins and Omega-3 fatty acids has been shown to significantly decrease the progression of the disease. Monitor closely, daily use of amsler grid. All questions were answered to the patient's satisfaction
### Response I do not know
Listing 11: OPT-IML-MAX-30B with guided instruction-based prompting: wrong prediction
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated
with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for
that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not
contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: coexists_with question: What does armd lead to? context: Acute Exudative ARMD OU - chronic
leakage OU. No significant changes OU. Recommend treating with Anti-VEGF injections . Discussed with patient the gravity of this disease, including the potential for vision loss and guarded recovery. Reviewed treatment options - Avastin, Lucentis, Eylea , PDT.
### Response I do not know
Listing 12: Vicuna-33B with instruction-based prompting: wrong prediction
Instruction: I want you to act as a medical question answering machine. I will provide you with questions and a context, and you will reply with the answers. If the answer is not in
context, answer "I do not know." question: What can decrease the chance of armd? context: Acute Exudative ARMD OS - Subertinal
blood/exudates OU-Discussed with patient the gravity of this disease, including the potential for vision loss and guarded recovery. Reviewed treatment options - Avastin, Lucentis, Eylea, PDT.
Answer: In the context provided, the question is about
decreasing the chance of ARMD (Age-Related Macular Degeneration). While I am not a
doctor, I can provide some general information on this topic.There are certain risk factors for ARMD that you may not be able to change, such as age, family history, and genetics. However, there are some lifestyle changes that may help decrease the risk of ARMD or slow its progression:Maintain a healthy diet rich in fruits, vegetables, and whole grains. Foods high in antioxidants, such as vitamins C and E, zinc, and lutein, may be particularly beneficial.Exercise regularly, as studies have shown that physical activity may reduce the risk of ARMD.Maintain a healthy weight, as obesity has been linked to a higher risk of ARMD.Quit smoking, as smoking is a risk factor for ARMD and can exacerbate the condition.Regularly check your vision and have annual eye exams, especially if you have a family history of ARMD or other risk factors. Early detection and treatment can help manage the condition more effectively.It is important to discuss these options with a healthcare professional to determine the best course of action for your specific situation.
Listing 13: LLama-2-70B-chat with instruction-based prompting: wrong prediction Instruction: I want you to act as a medical
question answering machine. I will provide you with
questions and a context, and you will reply with the answers. If the answer is not in context, answer "I do not know." question: What can heal amd? # TODO change heal to our question context: Wet ARMD (Inactive) OD - Improved, no active leakage was seen during dilated eye exam and diagnostic testing. Patient is advised to continue AREDS + WACS vitamins. Eat green leafy vegetables like Spinach 5 times a week and fish at least 2 times a week.Recommend close monitoring and treatment as indicated. Patient will return to office in 1 month for IVFA to monitor AMD . Answer: I cannot provide medical advice or recommend
treatments for specific individuals. However, I can provide some general information about age-related macular degeneration (AMD) and its treatment options.AMD is a chronic eye condition that affects the macula, the part of the retina responsible for central vision. There are two main types of AMD: dry and wet. Wet AMD is characterized by the growth of new blood vessels under the retina, which can leak fluid and cause vision loss.While there is no cure for AMD, there are several treatment options available to slow down its progression and manage its symptoms. For wet AMD, anti-vascular endothelial growth factor (VEGF) injections are commonly used to prevent further vision loss. These medications work by blocking the growth of new blood vessels and reducing the amount of fluid in the retina.In addition to medication, a healthy lifestyle can also help manage AMD. A diet rich in fruits, vegetables, and omega-3 fatty acids, along with regular exercise and not smoking, can reduce the risk of AMD progression.It’s important to note that the best course of treatment for AMD will depend on the individual patient and the severity of their condition. Regular eye exams and consultations with an eye care professional are necessary to determine the most appropriate treatment plan.In the context of the patient you described, the recommended treatment plan includes continuing AREDS + WACS vitamins, eating a healthy diet, and
close monitoring with regular eye exams. IVFA (intravitreal fusion) may also be recommended to monitor the progression of AMD and determine if additional treatment is needed. However, it’s important to consult with an eye care professional for personalized medical advice.
Listing 14: LLama-2-70B-chat with instruction-based prompting: wrong prediction
Instruction: I want you to act as a medical question answering machine. I will provide you with questions and a context, and you will reply with the answers. If the answer is not in
context, answer "I do not know." question: What can reduce the risk of armd? context: Acute Exudative ARMD OS - ICG/IVFA
shows active bleeding. Discussed with patient the gravity of this disease, including the potential for vision loss and guarded recovery. Reviewed treatment options - Avastin, Lucentis, Eylea, PDT.
Answer: I do not know.The question asked is not in
context with the given situation. The patient has acute exudative age-related macular degeneration (ARMD) with active bleeding, and the doctor has already discussed the severity of the disease and the potential treatment options with the patient. The question about what can reduce the risk of ARMD is not relevant to the current situation.
E.2.3 Examples of correct predictions
Listing 15: WizardLM-70B with guided instructionbased prompting: correct prediction
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for
that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: factor question: What can increase the risk of armd? context: 2 small Druse OD- clinically does not
look like ARMD. Patient has a family history of ARMD, recommend starting on AREDS + WACS vitamins. Eat green leafy vegetables like Spinach 5 times a week and fish at least 2 times a week.
### Response factor: factor: family history, diet, lifestyle
Listing 16: Vicuna-33B with guided instruction-based prompting: correct prediction
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: treat question: What can decrease the chance of armd? context: Acute Exudative ARMD OS - Subertinal
blood/exudates OU-Discussed with patient the gravity of this disease, including the potential for vision loss and guarded recovery. Reviewed treatment options - Avastin, Lucentis, Eylea, PDT.
### Response treat: Avastin, Lucentis, Eylea, PDT
Listing 17: LLama-2-70B-chat with guided instructionbased prompting: correct prediction
You are a helpful medical knowledge extractor assistant. Your task is to answer to the question, using only and only the provided context. Do not use any other sources. Do not use your knowledge. Answer to the question using the context. Do not write explanation. Do not write long sentences. Answer with extracted entities, separated with commas. I will provide the inputs in the following format
### Input question_type: treat/coexists_with/factor question: [QUESTION] context: [CONTEXT]
I want you to answer to the question in the following format, depending on the [ ENTITY_TYPE] which can be either [TREAT], [ COEXISTS_WITH], [FACTOR]. Do not include other entity types. Do not include treatment if not asked. Only extract entities for
that type.
### Response
[ENTITY_TYPE]: [ENTITY_1], [ENTITY_2], [ENTITY_3 ]...
If the question is not related to the context, or if the context contains a missing information to answer to the question, answer "I do not know." and nothing more. No explanations. If the context does not
contain the answer to the question, answer with the following format
### Response I do not know.
### Input
question_type: treat question: What can slow the progression of armd? context: Dry ARMD OU- Explained that there is no
specific treatment at this time. Can convert to wet ARMD, which is treatable. Eating a healthy diet that includes green leafy vegetables, fish, taking AREDS + WACS eye vitamins and Omega-3 fatty acids has been shown to significantly decrease the progression of the disease. Monitor closely. All questions were answered to the patient'
s satisfaction.
### Response treat: AREDS + WACS eye vitamins, Omega-3 fatty
acids, healthy diet including green leafy vegetables, fish
E.3 Encoder-decoder models E.3.1 Examples of wrong predictions
Listing 18: FLAN-UL2 with instruction-based few-shot prompting: wrong prediction Instruction: I want you to act as a question
answering machine. I will provide you with a question and a context, and you will reply with the answers.
Question: What can slow the progression of AMD? Context: Macular Dystrophy vs. Early Dry AMD OU -
Explained that there is no specific treatment at this time. Patient educated on condition. Eating a healthy diet that includes green leafy vegetables, fish, taking AREDS + WACS eye vitamins and Omega-3 fatty acids has been shown to significantly decrease the progression of the disease.
Answer: Eating a healthy diet that includes green leafy vegetables.
Question: What can slow the progression of myopic macular degeneration? Context: Myopic Macular Degeneration - Explained that there is no specific treatment at this time. AREDS and WACS vitamins MAY help slow down the progression of the degeneration. Monitor closely. All questions were answered to the patient's satisfaction. Answer: AREDS and WACS vitamins
Question: What can myopic macular degeneration convert to? Context: Myopic Macular Degeneration - Explained that there is no specific treatment at this time. AREDS and WACS vitamins MAY help slow down the progression of the degeneration. Monitor closely. All questions were answered to the patient's satisfaction. Answer: AREDS + WACS eye vitamins
E.3.2 Examples of correct predictions
Listing 19: FLAN-T5-XXL with instruction-based prompting: correct prediction I want you to act as a question answering
machine. I will provide you with a questions and a context and you will reply with the answers. Question: What can slow the progression of armd? Instruction: If the answer is not in context
answer "i do not know". Context: Wet ARMD- Patient educated on condition.
Discussed diet and use of Macular Degeneration vitamins. All questions were answered to the patient's satisfaction.
Answer: vitamins
Listing 20: FLAN-T5-XXL with few-shot prompting: correct prediction question: What can slow the progression of
macular disease? context: very strong association between smoking
and macular disease. Smokers are up to four times more likely than non-smokers to develop age-related macular degeneration ( AMD) and smokers with a genetic predisposition to AMD are eight times more likely to get the condition. Some research suggests people with certain genes can be 20 times more likely to get macular
degeneration if they smoke. target: the answer to the question given the
context is smoking.
question: What can slow the progression of amd? context: Macular Dystrophy vs. Early Dry AMD OU -
Explained that there is no specific treatment at this time. Patient educated on condition. Eating a healthy diet that includes green leafy vegetables, fish, taking AREDS + WACS eye vitamins and Omega-3 fatty acids has been shown to significantly decrease the progression of the disease. Stressed the need for follow up exams. All questions were answered to the patient's satisfaction.
target: the answer to the question given the context is Eating a healthy diet that includes green leafy vegetables.
question: What can slow the progression of myopic macular degeneration? context: D/w pt: Myopic macular degeneration. Diagnosis discussed with patient. Possible treatments explained including glasses, refractive surgery, contact lenses or doing nothing. All questions were answered to patients satisfaction. target: the answer to the question given the context is glasses
Listing 21: FLAN-UL2 with few-shot prompting: correct prediction very strong association between smoking and
macular disease. Smokers are up to four times more likely than non-smokers to develop age-related macular degeneration ( AMD) and smokers with a genetic predisposition to AMD are eight times more likely to get the condition. Some research suggests people with certain genes can be 20 times more likely to get macular
degeneration if they smoke. Create a bulleted list of what can slow the
progression of macular disease? - not smoking
Macular Dystrophy vs. Early Dry AMD OU - Explained that there is no specific treatment at this time. Patient educated on condition. Eating a healthy diet that includes green leafy vegetables, fish, taking AREDS + WACS eye vitamins and Omega-3 fatty acids has been shown to significantly decrease the progression of the disease.
Stressed the need for follow up exams. All questions were answered to the patient's satisfaction. target: the answer to the question given the context is Eating a healthy diet that includes green leafy vegetables.
Create a bulleted list of what can slow the progression of amd? - Eating a healthy diet - Green leafy vegetables
Myopic Macular Degeneration - Explained that there is no specific treatment at this time. AREDS and WACS vitamins MAY help slow down the progression of the degeneration. Monitor closely. All questions were answered to the patient's satisfacti Create a bulleted list of What can slow the progression of myopic macular degeneration?. - AREDS - WACS vitamins
Listing 22: FLAN-UL2 with instruction prompting: correct prediction Instruction: I want you to act as a medical
question answering machine. I will provide you with
questions and a context, and you will reply with the answers. Question: What does armd affect? Instruction: If the answer is not in context,
answer "I do not know." Context: Acute Exudative ARMD/ CSCR OD - appears
slightly worse on OCT and exam. Reviewed treatment options - Avastin, Lucentis, Eylea , PDT.
Answer: I do not know
F Implementation Details
All the method’s hyperparameters have been selected through experimentation with the data and may be adjusted for the specific dataset being utilized. Further explanation on the rationale behind the selection of each hyperparameter is provided in Table 6.

heading: Future Work
text: Fine-tuning LLMs on a relevant corpus could potentially enhance their performance and accuracy, and this remains an area for future exploration