Automatic Extraction of Disease Risk Factors from Medical Publications (2024)

Abstract: We present a novel approach to automating the identification of risk factors for diseases from medical literature, leveraging pre-trained models in the bio-medical domain, while tuning them for the specific task. Faced with the challenges of the diverse and unstructured nature of medical articles, our study introduces a multistep system to first identify relevant articles, then classify them based on the presence of risk factor discussions and, finally, extract specific risk factor information for a disease through a question-answering model. Our contributions include the development of a comprehensive pipeline for the automated extraction of risk factors and the compilation of several datasets, which can serve as valuable resources for further research in this area. These datasets encompass a wide range of diseases, as well as their associated risk factors, meticulously identified and validated through a fine-grained evaluation scheme. We conducted both automatic and thorough manual evaluation, demonstrating encouraging results. We also highlight the importance of improving models and expanding dataset comprehensiveness to keep pace with the rapidly evolving field of medical research.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 474–485 August 16, 2024. ©2024 Association for Computational Linguistics
474
Our contributions include the development of a comprehensive pipeline for the automated extraction of risk factors and the compilation of several datasets, which can serve as valuable resources for further research in this area. These datasets encompass a wide range of diseases, as well as their associated risk factors, meticulously identified and validated through a fine-grained evaluation scheme. We conducted both automatic and thorough manual evaluation, demonstrating encouraging results. We also highlight the importance of improving models and expanding dataset comprehensiveness to keep pace with the rapidly evolving field of medical research.

heading: 1 Introduction
text: Automatic identification of risk factors for diseases plays a pivotal role in preventive medicine, enabling healthcare professionals to formulate effective prevention strategies and improve patient outcomes. Traditionally, this process has relied heavily on manual review of extensive medical literature, a time-consuming and labor-intensive task, hindering knowledge accessibility and effective usage.
As a concrete example, recently, compelling evidence has emerged linking Lipoprotein A (Lp(a)) — a particle operating similarly to the more familiar LDL molecule — to the pathogenesis of atherosclerosis and subsequent coronary artery disease, commonly referred to as Myocardial Infarction (MI).
Despite the established role of Lp(a) as a risk factor (Kronenberg et al., 2022), many primary care clinicians remain inadequately informed, occasionally lacking knowledge regarding its testing procedures. Moreover, in a conversation with a board-certified professor of interventional cardiology, he disclosed receiving frequent inquiries from other clinicians questioning the necessity of referrals for Lp(a) testing. This highlights the pressing need for an automated tool capable of screening vast amounts of scientific literature and identifying prominent risk factors for various diseases.
Despite significant advances in the field of natural language processing, automatic extraction of disease risk factors from scientific medical literature remains a challenging endeavor. Contrary to the analysis of electronic health records (Chen et al., 2015; Boytcheva et al., 2017; Chokwijitkul et al., 2018), here the primary challenge lies in the diverse and unstructured nature of medical publications, where risk factors are described in various contexts and formats. What is more, the continuous discovery of new risk factors necessitates a dynamic approach that can adapt to the evolving body of medical knowledge. This study introduces a novel approach to automating the identification of disease risk factors from medical literature.
Utilizing pre-trained large language models, based on BioBERT (Lee et al., 2020), we developed a multi-step system, that first identifies relevant medical articles, classifies them based on the presence of risk factor discussions, and then extracts specific risk factor information through a question answering (QA) model. Our approach to extraction of disease risk factors is illustrated in Figure 1: (1) medical abstracts are retrieved from PubMed, (2) a specifically fine-tuned binary classifier is used to identify abstracts with risk factors information, and (3) textual spans containing risk factors are extracted via a question answering model, fine-tuned on manually annotated QA items.
The contribution of this work, therefore, twofold: First, we present a comprehensive pipeline for automated extraction of risk factors. Second, we compile and make available several datasets that can serve as valuable resources for future research in this field. These datasets include a carefully annotated, large and diverse set of over 1,700 risk factors associated with 15 diseases, as well as set of over 160,000 automatically extracted risk factors,1 with almost 1,500 manually assessed for their quality, using a fine-grained annotation scheme.2
We survey the related work in Section 2 and detail on collection and annotation of our datasets in Section 3. We next describe our approach to the task and report experimental results in Section 4. Human evaluation results are presented in Section 5. Discussion of the difficulty of the task and the limitations of this work are presented in Section 6. We conclude this study in Section 7.

heading: 2 Related Work
text: Automatic identification of disease risk factors through the analysis of medical texts has garnered interest across various research domains, particularly in applying natural language processing and machine learning techniques to electronic health records (EHRs) and electronic medical records (EMRs). Here we review key contributions in this area, highlighting approaches that parallel and diverge from our focus on free-text medical articles.
Chokwijitkul et al. (2018) explore the utilization of deep learning models to extract heart disease
1We note that the set of over 160,000 automatically extracted risk factors are of admittedly mixed quality (see Section 5 and Table 5 for details), yet, we thought this data can serve the community for further research in the field.
2All code and data are available at https://github.com/ maximrub/diseases-risk-factors.
risk factors from EHRs. The approach, grounded in analyzing structured data within EHRs, contrasts with our exploration of unstructured text in medical literature, underscoring the diversity in data sources for risk factor identification. Boytcheva et al. (2017) attempt at mining clinical texts for risk factor identification using association rules. Specifically, they handle data in XML format from the Diabetes Register, indicating a structured approach to data analysis. This work differs from ours in terms of both data source type (clinical narratives), as well as in our broader application to unstructured, free-text medical articles and the use of pre-trained large language models (LLMs) for the task of text understanding.
A comprehensive work on identifying risk factors for heart disease (from clinical data) over time was done in a shared task organized by UTHealth3 (Stubbs et al., 2015). Sheikhalishahi et al. (2019) offer an overview of NLP applications in analyzing clinical notes for chronic disease management, highlighting the increasingly significant contribution of language models to healthcare applications. In the domain of precision medicine, Sabra et al. (2017) focus on extracting semantic information and assessing sentiments in clinical notes.
Various works have employed data mining and machine learning (ML) techniques for identifying risk factors from patient data (Abdelhamid et al., 2023), or clinical outcome prediction (Kavakiotis et al., 2017; Mehmood et al., 2021; Naik et al., 2021). Recently, the identification of risk factors for delirium prediction, a rare adverse reaction observed in COVID-19 patients, was developed utilizing ML applied to nursing records (Miyazawa et al., 2024). Additional line of studies focuses
3The University of Texas Health Science Center.
on building language models specifically-tailored for medical literature related tasks (Roitero et al., 2021; Yang et al., 2022; Singhal et al., 2023).
Several significant contributions have been made in the field of biomedical relation extraction, which includes identifying factors that predispose individuals to diseases. The SemRep (Kilicoglu et al., 2020) tool extracts semantic predications from biomedical texts, including relationships such as "predisposes". The outputs of SemRep have been used to create SemMedDB (Kilicoglu et al., 2012), a large-scale repository of semantic predications from PubMed. Building on these resources, BioPREP (Hong et al., 2021) employs deep learning techniques for predicate classification. The BioRED (Luo et al., 2022) dataset includes a "positive correlation" relation between diseases and other biomedical entities like genes and chemicals.
Conclusion While the majority of existing research focuses on analyzing structured electronic health records and electronic medical records to identify disease risk factors, our study pushes beyond these confines by examining free-text medical literature. Processing unstructured medical text introduces distinct challenges, especially due to language complexity, variation, and the potential for nuanced double meanings, and even worse, due to the necessity to discern context accurately. Consequently, it opens up expansive opportunities for subtle understandings of disease risk factors, facilitating both research and practical applications.

heading: 3 Dataset
text: Data collection process for this work can be viewed as a three-step process: (i) collection of the set of disease names spanning multiple disease families, (ii) manual annotation of scientific article abstracts containing explicit mention of risk factors of a subset of diseases – "abstracts seed", and (iii) manual annotation of risk factors description (span) in abstract texts found in (ii) – "risk factors seed". We detail on each step in this multi-phase procedure.

heading: 3.1 Disease Dataset Collection
text: Aiming to assemble a comprehensive list of diseases, we made use of the KEGG Disease Database API4 to retrieve disease-related information, including names, description and relevant medical codes
4KEGG database: https://www.kegg.jp/kegg/ disease/; specifically, we used its REST API service at https://www.kegg.jp/kegg/rest/ for retrieval.
such as MeSH (Medical Subject Headings), ICD10 and ICD-11.5 This process resulted in 2,624 distinct disease names, comprising the foundation for further retrieval of scientific abstracts and, ultimately, automatic extraction of risk factors, from scientific medical literature.

heading: 3.2 Seed Dataset with Relevant Abstracts
text: Retrieval of Abstracts Discussing Risks Using the list of disease names retrieved from KEGG, we next queried PubMed6 — a large, reliable, and authoritative resource of biomedical literature — for article abstracts containing the disease names. Specifically, we used the Entrez Programming Utilities7 via the biogo package.8 The inherent limitation of this study is related to the fact that only abstracts are freely available through the PubMed interface. However, paper abstracts typically contain a concise summary and main findings of the work, hence constitute a sufficient input for the task at hand. Similarly, prior studies analyzed abstracts retrieved from PubMed for building a biological network (Chen and Sharp, 2004), topical clustering (David and Samuel, 2012), and identification of negative and positive domain-specific medical terms (Vinkers et al., 2015).
Aiming at retrieval of abstracts discussing findings related to risk factors, we queried PubMed for containment of the phrase "risk factor" in a paper’s information: title, abstract or MeSH terms. The following pseudo-code was used for this purpose:
where disease_name refers to the disease we are seeking risk factors for, and the exact search term "risk factor" (surfacting also the plural "risk factors") can appear in abstract, title or MeSH terms.
Annotation of Abstracts for Risk Factors Despite the evident potential, not every abstract with explicit mention of "risk factor" or marked with a "risk factor" MeSH term contains risk factors for a pre-defined disease. As a concrete example, a medical study can mention a list of potential risk factors tested, without any of them showing as significant.
5As of April 2024, ICD-11 (International Classification of Diseases, v11) is the most up-to-date code collection.
6https://pubmed.ncbi.nlm.nih.gov 7https://www.ncbi.nlm.nih.gov/books/NBK25501 8https://github.com/biogo/ncbi
We, therefore, define our first (pre-processing) task as automatic classification of a retrieved abstract for spelling out an artifact, found to be a risk factor for the disease in the study.
A qualified annotator with medical background (one of the authors of this paper) annotated a random set of 182 abstracts. The procedure resulted in 87 positive abstracts (explicitly mentioning a risk factor) and 95 negative, thereby comprising a sufficient training set for the binary classifier – step (2) in the pipeline in Figure 1. Table 1 shows two examples of relevant abstract parts containing riskrelated phrases which do or do not qualify as risk factors, as identified by the annotator. Evidently, the nuanced language used to discuss risks in various contexts renders the task as non-trivial for both humans and automatic tools.

heading: 3.3 QA Seed Dataset with Risk Factors
text: Given an article abstract specifying a risk factor(s) for a certain disease, we cast the risk factor identification problem as extractive question answering scenario, where given the abstract and the question "What are the risk factors for {disease name}?", a textual span, containing the answer, will be identified. In Section 4.1.2 we make use of the established and popular BERT-based QA model – BioBERT9 (Lee et al., 2020), and fine-tune it for the task at hand using a manually annotated set of QA items: context (article abstract), a targeted question of the form mentioned above, and a set of manually marked answers in the form span_start and answer_text (implying span_end).
In the absence of suitable annotated datasets for this nuanced task, we developed a web interface for medical students to manually annotate article abstracts. This interface is used for (manual) identification of text segments within abstracts, given the disease discussed in the article. We present a few screenshots of the annotating tool in Appendix A.1, and release the tool for the community.
The annotator with medical background marked text spans containing risk factors in a random set of 668 abstracts identified to contain explicit mention of a risk factor,10. resulting in the total of 1,712 QA items, spanning 15 diverse diseases,11 where each QA item reflects a single risk factor in an abstract that (possibly) encompasses multiple valid risks.
9https://huggingface.co/dmis-lab/biobert-v1.1 10The abstracts were sampled from the set automatically
classified as "positive" (see Section 4.1.1) 11Appendix B reports the full list of diseases.
Sentences suggesting risk factors significant only within specific population subgroups were denoted as such. Table 4 presents two examples of QA items: disease name, abstract, and the highlighted risk factor span, as marked by the annotator.
Collectively the carefully-curated and annotated set of abstracts for binary classification of medical articles, and the set of QA items, comprise a highquality collection for tuning pre-trained language models for the purpose of this study.

heading: 4 Methodology and Experiments
text: We further describe in detail our methodological approach, experimental setup and results.

heading: 4.1 Methodology
text: As illustrated in Figure 1, we apply a multi-step approach to automate the identification of disease risk factors from medical literature. Central to our methodology is the use of BioBERT, a variant of BERT pre-trained on biomedical texts, enabling nuanced understanding of complex medical language (Lee et al., 2020). We next provide details on each step in the process. This model was chosen due to its proven benefits in the biological domain, and its encoder-based architecture – (arguably) the most appropriate choice for both the classification and extractive question answering tasks at hand.12

heading: 4.1.1 Detection of Abstracts with Risk Factors
text: The pre-trained BioBERT-based classifier13 was tuned for abstracts classification using the training part (80%) out of over 182 manually annotated abstracts (see Section 3.2), and tested on the held-out part (20%), achieving the accuracy of 92%. Table 3 reports the per-class classification results. This encouraging result facilitated our efforts of analyzing content that is most likely to yield valuable insights into disease-risk factor associations.
We collected a substantial dataset of abstracts, by querying PubMed for each one of over 2400 diseases, as detailed in Section 3.2; this step resulted in 137,740 abstracts. We next apply the fine-tuned classifier to identify abstract potentially containing risk factors for a disease. Out of the total number of 137,740 abstracts, 89,834 were classified as positive – containing explicit mentions of risk
12Our future work includes investigation of decoder-based models (e.g., GPT), casting the QA part as an abstractive task.
13https://huggingface.co/dmis-lab/biobert-v1.1. We used the default settings with max_input_length of 512 tokens, training the classifier for three epochs.
factors for diseases. Naturally, some diseases (and disease families) resulted in more prolific retrieval, due to their higher coverage in the medical literature: while various cancer types (e.g., Carcinoma, Leukemia) have large body of related articles, genetic disorders are surveyed less frequently in the context of risk factor discussion.

heading: 4.1.2 Identification of Disease Risk Factors
text: The collection of abstracts classified positively to contain a risk factor, was then subject to the task of risk factor extraction – step (3) in Figure 1. We cast the task as extractive QA, where the medical abstract represents the context, and the question template is formulated as "What are the risk factors for {disease name}?". We anticipate the BioBERT QA model (Lee et al., 2020) to identify span(s) in the abstract containing the answer (or answers, in case multiple risk factors are mentioned in the same abstract), similarly to examples presented in Table 2. We fine-tune the model for the specific task, as described below.
Fine-tuning the QA Model We tuned the BioBERT model for our usecase using the training part (80%) of the 1,712 QA items annotated manually by the author with medical background
(see Section 3.3); the remaining 20% were used for testing. Notably, the set of 15 diseases in the 668 abstracts was carefully split into training and test sets, so that the same disease does not appear in both sets, facilitating the assessment of the model’s generalizability and performance across a variety of disease contexts. The model tuning was done using the maximum context length of 384 tokens, learning rate of 2e-5, and 25 epochs.
We use two common metrics for automatic evaluation of extractive question answering: exact-match and F1-score. Applied on the test set (342 QA items), the metrics obtained 61.76% for exact-match, and 88.23% for F1-score, highlighting the potential of the approach.
Determining the Maximum Answer Length We determined the maximum length for answers in our QA model by analyzing the lengths of all answers within our training dataset. We calculated the length of each answer (in characters) and studied their distribution. The maximum answer length was set at the 95th percentile of these lengths to encompass the majority of real-world answers while excluding outliers. This threshold is crucial for maintaining focus on concise and relevant answer segments, thereby enhancing the model’s training
and operational effectiveness. In practice, when the model evaluates potential answers, it only considers text segments whose length does not exceed this predefined limit. Specifically, the text extracted between the predicted start and end indices is compared against the maximum length, and any text exceeding this threshold is disregarded.
Identification of Risk Factors at Scale Utilizing the fine-tuned QA model, we then processed the collected abstracts to identify and validate risk factors for a wide range of diseases, culminating in a dataset that catalogs these findings in much detail. As a concrete example, the entry for the "B-cell acute lymphoblastic leukemia" includes 16 (not necessarily unique) automatically extracted risk factors. Along with the extracted span, the BioBERT QA model provides its probability (confidence, in the 0-1 range) for the identified answer. For a given disease, we only considered answers exceeding the confi-
dence of 0.6*max_answer_probability, where the max_answer_probability is the maximum probability assigned to an answer for the disease. The final dataset encompasses the total of 162,409 identified risk factors spanning 744 diseases, extracted from 54,820 PubMed abstracts.
Due to the inherently strict nature of the exact-match metric, we could observe multiple cases where the extracted answer was largely correct, but didn’t represent a precise overlap with the "gold" answer due to a single missing or redundant word. In particular, while some cases surface useful information about a disease risk factors, they are marked as inaccurate by the automatic metric. We complement the evaluation pipeline by sampling a large amount of (automatically identified) risk factors for diseases, and performing fine-grained human assessment of the results’ quality.

heading: 5 Human Evaluation
text: We next manually evaluated a random sample of 1,485 extracted risk factors spanning 29 various diseases (constituting roughly 1% of the full set of extracted factors), based on their validity and relevance to the disease in question.

heading: 5.1 Evaluation Scheme
text: We designed a specifically-tailored, four-tiered annotation scheme for the sake of reliable and accurate evaluation, as detailed below. Each risk factor was scored with one of three annotation marks, following the below annotation scheme:
(1) Valid risk factor for the specified disease: Correctly identified risk factor extracted for the disease of interest, i.e., the disease in the question introduced to the QA system.
(2) Valid risk factor for a different disease: Correctly identified risk factor for a different disease, i.e., not the disease in the question introduced to the QA system, indicating capabilities yet highlighting challenges in specificity.
(3) Invalid risk factor: Phrases and terms that are not considered medical risk factors.
Additional distinction was done within the first group (valid risk factor), annotating risk factors with strong statistical correlation, as evident from the abstract by inspecting statistical measurements as odd ratio (OR), and confidence intervals (CIs) – metrics often used in medical literature for testing the significance of findings, such as the presence of a factor in one population but not the other. 41 out of the total of 1,485 were marked as highly significant risk factors; we release these annotations as well to facilitate further research in the community.

heading: 5.2 Evaluation Results
text: Table 4 presents error analysis of correctly- and incorrectly-identified risk factor examples (the first two rows), as well as an example for artifact that does not constitute a risk factor (the last row).
We attribute most factors erroneously annotated with type 3 annotation — not a risk factor — to cases where the QA model was required to extract a risk factor from an abstracts that does not contain one. Since the model was trained (and fine-tuned) to always identify an answer span for a given context and question, it is expected to yield (admittedly) weak performance on a context lacking the factors at the first place. Notably, a relatively small amount of all manually evaluated examples (around 8.5%) fall into this category.
Table 5 further summarizes the evaluation results by disease family. The prevalence of type 1 and 2 annotations illustrates the model’s effectiveness in identifying risk factors, yet also underscores
the challenges in achieving precise disease-specific accuracy. The presence of type 3 annotations, although significantly lower, highlights the ongoing need for the classification model refinement to enhance both specificity and accuracy.
Error Analysis Additional observation can be made about error distribution between type 1 and 2 annotations within and across disease families. Evidently, while some disease families show a balanced ratio between type 1 and 2 annotations (e.g., Infection, Leukemias), others resulted in more mis-identified factors – type 2 annotation (e.g., Metabolic disorders). We hypothesize that abstracts concerning diseases with a significant, sometimes absolute, genetic component are less likely to address other contributing factors. Consequently, research in this area predominantly focuses on stratifying potential risks for other diseases in individuals already affected by the genetic disorder.

heading: 6 Discussion and Limitations
text: Our study, while contributing valuable insights into the automation of risk factor identification from medical publications, is subject to several limitations that merit a thorough discussion.
One of the primary limitations is the challenge of accurately distinguishing risk factors specifically associated with the disease in question (type 1) from valid risk factors that are not directly related to the disease under investigation (type 2). While our models demonstrated a high capacity for identifying potential risk factors, the precision in contextualizing these factors to specific diseases varied. This aspect highlights a critical area for future research, emphasizing the need for enhanced specificity in the models to improve their utility in targeted medical research and practice.
Moreover, the study’s reliance on free-text medical articles introduces variability in the data quality and representation. The unstructured nature of these texts and the diversity in how risk factors are described pose significant challenges for both the binary classification and question-answering models. Efforts to standardize data representation and improve model robustness against such variability are essential steps forward.
The datasets used in this study, while extensive, are not exhaustive. The landscape of medical research is continuously evolving, with new findings emerging regularly. The datasets, therefore, rep-
resent a snapshot in time, and ongoing efforts to update and expand these resources are necessary to maintain their relevance and utility.
Finally, the study’s scope was constrained by the computational resources available. Future work could explore more complex models or ensemble approaches that might offer improved accuracy but require more substantial computational power.
Despite these limitations, this study represents a significant step toward automating the identification of disease risk factors from medical literature. Acknowledging and addressing these limitations in future research will be crucial for advancing the field and enhancing the practical applicability of these technologies in healthcare.

heading: 7 Conclusions and Future Work
text: This study presented an approach to identifying and extracting disease risk factors from free-text medical articles using advanced natural language processing techniques, specifically leveraging the capabilities of the pre-trained BioBERT-based architecture. Our methodology involved a multi-step process, including the retrieval of relevant articles, binary classification to filter articles discussing risk factors, and a question-answering model to extract specific risk factor information.
We have demonstrated the potential of language technologies to significantly enhance the efficiency and effectiveness of risk factor identification in medical literature. Our contributions to this field are twofold: the presentation of an automated
pipeline for risk factor extraction and the creation of valuable datasets for future research. While our study marks an advancement in the automated extraction of risk factors from medical literature, there remain several avenues for future research and development. Our future directions include introducing improvements to QA model’s accuracy and specificity, integration of additional data sources, and evaluation of more advanced LLMs for the task of risk factors identification.
Furthermore, inspired by recent findings that automatic annotations generated by models like GPT-4 can achieve results comparable to human annotations, we plan to investigate the use of GPT-4 for the task of risk factors annotation, and compare its performance with human experts.

heading: 8 Ethical Considerations
text: We make use of publicly available data in the domain of healthcare, that have been broadly used in numerous studies. Manual annotations were conducted by one of the authors of the paper, with medical background. Due to the required expertise and the inherent difficulty of the task, the mean hourly rate for the annotator was much higher than the established minimum wage.

heading: Acknowledgements
text: We are grateful to Noga Shraibman for much help creating datasets during the early stages of this work. We are also thankful to our three anonymous reviewers for their constructive feedback.

heading: A Appendices
text: A.1 Overview of the Risk Factor Annotation System Architecture
The risk factor annotation system comprises three main components designed to streamline the process of annotating risk factors in medical articles. This system was instrumental in creating the datasets used in our research.
GraphQL Server The backbone of the system is a GraphQL server, which serves as the central communication hub. Hosted on Kubernetes (k8s) for scalability and reliability, the server facilitates data exchange between the user interface and the database. It handles requests for data retrieval and submission, ensuring that the web application and the code can access and store data efficiently.
Web UI The front end of the system is a Reactbased web application, also deployed on Kubernetes for high availability. This intuitive user interface allows medical students and researchers to interact with the system, including retrieving medical articles, annotating risk factors within texts, and submitting these annotations back to the server. The design prioritizes ease of use to facilitate accurate and efficient annotation work.
Python Algorithm Complementing the user interface is a Python-based algorithm that interacts with the GraphQL server. This component is responsible for processing medical articles, including sending requests to the server to fetch articles for annotation and submitting the results of automated risk factor identification processes. It plays a critical role in pre-processing and post-processing steps in the dataset creation pipeline.
Database At the core of the system lies a MongoDB database hosted on Azure Cosmos DB. This NoSQL database was chosen for its scalability, flexibility, and robust support for storing unstructured data, such as medical article texts and annotations. It stores all data related to diseases, articles, and user annotations, providing a persistent and reliable data storage solution for the system.
Figures 2-3 illustrate two screenshots of the application developed for manual annotation of risk factors. The system code will also be made available per acceptance.

heading: B Diseases with Annotated Risk Factors in the QA dataset (the training set)
text: Section 3.3 details the procedure of manual annotation of risk factors following the step of abstract retrieval. The annotated data comprises 1,712 QA items from 668 abstracts covering 15 diseases from multiple disease families, as detailed in Table 6.

heading: C Diseases with Evaluated Risk Factors
text: Table 7 reports the distribution of manually evaluated risk factors by disease family.

heading: Future Work
text: Our contributions to this field are twofold: the presentation of an automated
pipeline for risk factor extraction and the creation of valuable datasets for future research. While our study marks an advancement in the automated extraction of risk factors from medical literature, there remain several avenues for future research and development. Our future directions include introducing improvements to QA model’s accuracy and specificity, integration of additional data sources, and evaluation of more advanced LLMs for the task of risk factors identification. Furthermore, inspired by recent findings that automatic annotations generated by models like GPT-4 can achieve results comparable to human annotations, we plan to investigate the use of GPT-4 for the task of risk factors annotation, and compare its performance with human experts