GenAI4ELab/papercli-papers huggingface dataset:
REAL: A Retrieval-Augmented Entity Linking Approach for Biomedical Concept Recognition
Authors: Darya Shlyk; Tudor Groza; Marco Mesiti; Stefano Montanelli; Emanuele Cavalleri
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Large Language Models (LLMs) offer an appealing alternative to training dedicated models for many Natural Language Processing (NLP) tasks. However, outdated knowledge and hallucination issues can be major obstacles in their application in knowledge-intensive biomedical scenarios. In this study, we consider the task of biomedical concept recognition (CR) from unstructured scientific literature and explore the use of Retrieval Augmented Generation (RAG) to improve accuracy and reliability of the LLM-based biomedical CR. Our approach, named REAL (Retrieval Augmented Entity Linking), combines the generative capabilities of LLMs with curated knowledge bases to automatically annotate natural language texts with concepts from bio-ontologies. By applying REAL to benchmark corpora on phenotype concept recognition, we show its effectiveness in improving LLM-based CR performance. This research highlights the potential of combining LLMs with external knowledge sources to advance biomedical text processing.
Is That the Right Dose? Investigating Generative Language Model Performance on Veterinary Prescription Text Analysis
Authors: Brian Hur; Lucy Lu Wang; Laura Hardefeldt; Meliha Yetisgen
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Optimizing antibiotic dosing recommendations is a vital aspect of antimicrobial stewardship (AMS) programs aimed at combating antimicrobial resistance (AMR), a significant public health concern, where inappropriate dosing contributes to the selection of AMR pathogens. A key challenge is the extraction of dosing information, which is embedded in free-text clinical records and necessitates numerical transformations. This paper assesses the utility of Large Language Models (LLMs) in extracting essential prescription attributes such as dose, duration, active ingredient, and indication. We evaluate methods to optimize LLMs on this task against a baseline BERT-based ensemble model. Our findings reveal that LLMs can achieve exceptional accuracy by combining probabilistic predictions with deterministic calculations, enforced through functional prompting, to ensure data types and execute necessary arithmetic. This research demonstrates new prospects for automating aspects of AMS when no training data is available.
MiDRED: An Annotated Corpus for Microbiome Knowledge Base Construction
Authors: William Hogan; Andrew Bartko; Jingbo Shang; Chun-Nan Hsu
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: The interplay between microbiota and diseases has emerged as a significant area of research facilitated by the proliferation of cost-effective and precise sequencing technologies. To keep track of the many findings, domain experts manually review publications to extract reported microbe-disease associations and compile them into knowledge bases. However, manual curation efforts struggle to keep up with the pace of publications. Relation extraction has demonstrated remarkable success in other domains, yet the availability of datasets supporting such methods within the domain of microbiome research remains limited. To bridge this gap, we introduce the Microbe-Disease Relation Extraction Dataset (MiDRED); a human-annotated dataset containing 3,116 annotations of fine-grained relationships between microbes and diseases. We hope this dataset will help address the scarcity of data in this crucial domain and facilitate the development of advanced text-mining solutions to automate the creation and maintenance of microbiome knowledge bases.
Do Numbers Matter? Types and Prevalence of Numbers in Clinical Texts
Authors: Rahmad Mahendra; Damiano Spina; Lawrence Cavedon; Karin Verspoor
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: In this short position paper, we highlight the importance of numbers in clinical text. We first present a taxonomy of number variants. We then perform corpus analysis to analyze characteristics of number use in several clinical corpora. Based on our findings of extensive use of numbers, and limited understanding of the impact of numbers on clinical NLP tasks, we identify the need for a public benchmark that will support investigation of numerical processing tasks for the clinical domain.
A Fine-grained citation graph for biomedical academic papers: the finding-citation graph
Authors: Yuan Liang; Massimo Poesio; Roonak Rezvani
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Citations typically mention findings as well as papers. To model this richer notion of citation, we introduce a richer form of citation graph with nodes for both academic papers and their findings: the finding-citation graph (FCG). We also present a new pipeline to construct such a graph, which includes a finding identification module and a citation sentence extraction module. From each paper, it extracts rich basic information, abstract, and structured full text first. The abstract and vital sections, such as the results and discussion, are input into the finding identification module. This module identifies multiple findings from a paper, achieving an 80% accuracy in multiple findings evaluation. The full text is input into the citation sentence extraction module to identify inline citation sentences and citation markers, achieving 97.7% accuracy. Then, the graph is constructed using the outputs from the two modules mentioned above. We used the Europe PMC to build such a graph using the pipeline, resulting in a graph with 14.25 million nodes and 76 million edges.
Evaluating Large Language Models for Predicting Protein Behavior under Radiation Exposure and Disease Conditions
Authors: Ryan Engel; Gilchan Park
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: The primary concern with exposure to ionizing radiation is the risk of developing diseases. While high doses of radiation can cause immediate damage leading to cancer, the effects of low-dose radiation (LDR) are less clear and more controversial. To further investigate this, it necessitates focusing on the underlying biological structures affected by radiation. Recent work has shown that Large Language Models (LLMs) can effectively predict protein structures and other biological properties. The aim of this research is to utilize open-source LLMs, such as Mistral, Llama 2, and Llama 3, to predict both radiation-induced alterations in proteins and the dynamics of protein-protein interactions (PPIs) within the presence of specific diseases. We show that fine-tuning these models yields state-of-the-art performance for predicting protein interactions in the context of neurodegenerative diseases, metabolic disorders, and cancer. Our findings contribute to the ongoing efforts to understand the complex relationships between radiation exposure and disease mechanisms, illustrating the nuanced capabilities and limitations of current computational models.
XrayGPT: Chest Radiographs Summarization using Large Medical Vision-Language Models
Authors: Omkar Chakradhar Thawakar; Abdelrahman M. Shaker; Sahal Shaji Mullappilly; Hisham Cholakkal; Rao Muhammad Anwer; Salman Khan; Jorma Laaksonen; Fahad Khan
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: The latest breakthroughs in large language models (LLMs) and vision-language models (VLMs) have showcased promising capabilities toward performing a wide range of tasks. Such models are typically trained on massive datasets comprising billions of image-text pairs with diverse tasks. However, their performance on task-specific domains, such as radiology, is still under-explored. While few works have recently explored LLMs-based conversational medical models, they mainly focus on text-based analysis. In this paper, we introduce XrayGPT, a conversational medical vision-language (VLMs) model that can analyze and answer open-ended questions about chest radiographs. Specifically, we align both medical visual encoder with a fine-tuned LLM to possess visual conversation abilities, grounded in an understanding of radiographs and medical knowledge. For improved alignment of chest radiograph data, we generate ~217k interactive and high-quality summaries from free-text radiology reports. Extensive experiments are conducted to validate the merits of XrayGPT. To conduct an expert evaluation, certified medical doctors evaluated the output of our XrayGPT on a test subset and the results reveal that more than 70% of the responses are scientifically accurate, with an average score of 4/5. We hope our simple and effective method establishes a solid baseline, facilitating future research toward automated analysis and summarization of chest radiographs. Code, models, and instruction sets will be publicly released.
Multilevel Analysis of Biomedical Domain Adaptation of Llama 2: What Matters the Most? A Case Study
Authors: Vicente Ivan Sanchez Carmona; Shanshan Jiang; Takeshi Suzuki; Bin Dong
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Domain adaptation of Large Language Models (LLMs) leads to models better suited for a particular domain by capturing patterns from domain text which leads to improvements in downstream tasks. To the naked eye, these improvements are visible; however, the patterns are not so. How can we know which patterns and how much they contribute to changes in downstream scores? Through a Multilevel Analysis we discover and quantify the effect of text patterns on downstream scores of domain-adapted Llama 2 for the task of sentence similarity (BIOSSES dataset). We show that text patterns from PubMed abstracts such as clear writing and simplicity, as well as the amount of biomedical information, are the key for improving downstream scores. Also, we show how another factor not usually quantified contributes equally to downstream scores: choice of hyperparameters for both domain adaptation and fine-tuning.
Mention-Agnostic Information Extraction for Ontological Annotation of Biomedical Articles
Authors: Oumaima El Khettari; Noriki Nishida; Shanshan Liu; Rumana Ferdous Munne; Yuki Yamagata; Solen Quiniou; Samuel Chaffron; Yuji Matsumoto
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Biomedical information extraction is crucial for advancing research, enhancing healthcare, and discovering treatments by efficiently analyzing extensive data. Given the extensive amount of biomedical data available, automated information extraction methods are necessary due to manual extraction’s labor-intensive, expertise-dependent, and costly nature. In this paper, we propose a novel two-stage system for information extraction where we annotate biomedical articles based on a specific ontology (HOIP). The major challenge is annotating relation between biomedical processes often not explicitly mentioned in text articles. Here, we first predict the candidate processes and then determine the relationships between these processes. The experimental results show promising outcomes in mention-agnostic process identification using Large Language Models (LLMs). In relation classification, BERT-based supervised models still outperform LLMs significantly. The end-to-end evaluation results suggest the difficulty of this task and room for improvement in both process identification and relation classification.
Automatic Extraction of Disease Risk Factors from Medical Publications
Authors: Maxim Rubchinsky; Ella Rabinovich; Adi Shribman; Netanel Golan; Tali Sahar; Dorit Shweiki
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: We present a novel approach to automating the identification of risk factors for diseases from medical literature, leveraging pre-trained models in the bio-medical domain, while tuning them for the specific task. Faced with the challenges of the diverse and unstructured nature of medical articles, our study introduces a multi-step system to first identify relevant articles, then classify them based on the presence of risk factor discussions and, finally, extract specific risk factor information for a disease through a question-answering model. Our contributions include the development of a comprehensive pipeline for the automated extraction of risk factors and the compilation of several datasets, which can serve as valuable resources for further research in this area. These datasets encompass a wide range of diseases, as well as their associated risk factors, meticulously identified and validated through a fine-grained evaluation scheme. We conducted both automatic and thorough manual evaluation, demonstrating encouraging results. We also highlight the importance of improving models and expanding dataset comprehensiveness to keep pace with the rapidly evolving field of medical research.
Intervention extraction in preclinical animal studies of Alzheimer’s Disease: Enhancing regex performance with language model-based filtering
Authors: Yiyuan Pu; Kaitlyn Hair; Daniel Beck; Mike Conway; Malcolm MacLeod; Karin Verspoor
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: We explore different information extraction tools for annotation of interventions to support automated systematic reviews of preclinical AD animal studies. We compare two PICO (Population, Intervention, Comparison, and Outcome) extraction tools and two prompting-based learning strategies based on Large Language Models (LLMs). Motivated by the high recall of a dictionary-based approach, we define a two-stage method, removing false positives obtained from regexes with a pre-trained LM. With ChatGPT-based filtering using three-shot prompting, our approach reduces almost two-thirds of False Positives compared to the dictionary approach alone, while outperforming knowledge-free instructional prompting.
Efficient Biomedical Entity Linking: Clinical Text Standardization with Low-Resource Techniques
Authors: Akshit Achara; Sanand Sasidharan; Gagan N
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Clinical text is rich in information, with mentions of treatment, medication and anatomy among many other clinical terms. Multiple terms can refer to the same core concepts which can be referred as a clinical entity. Ontologies like the Unified Medical Language System (UMLS) are developed and maintained to store millions of clinical entities including the definitions, relations and other corresponding information. These ontologies are used for standardization of clinical text by normalizing varying surface forms of a clinical term through Biomedical entity linking. With the introduction of transformer-based language models, there has been significant progress in Biomedical entity linking. In this work, we focus on learning through synonym pairs associated with the entities. As compared to the existing approaches, our approach significantly reduces the training data and resource consumption. Moreover, we propose a suite of context-based and context-less reranking techniques for performing the entity disambiguation. Overall, we achieve similar performance to the state-of-the-art zero-shot and distant supervised entity linking techniques on the Medmentions dataset, the largest annotated dataset on UMLS, without any domain-based training. Finally, we show that retrieval performance alone might not be sufficient as an evaluation metric and introduce an article level quantitative and qualitative analysis to reveal further insights on the performance of entity linking methods.
XAI for Better Exploitation of Text in Medical Decision Support
Authors: Ajay Madhavan Ravichandran; Julianna Grune; Nils Feldhus; Aljoscha Burchardt; Roland Roller; Sebastian Möller
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: In electronic health records, text data is considered a valuable resource as it complements a medical history and may contain information that cannot be easily included in tables. But why does the inclusion of clinical texts as additional input into multimodal models, not always significantly improve the performance of medical decision-support systems? Explainable AI (XAI) might provide the answer. We examine which information in text and structured data influences the performance of models in the context of multimodal decision support for biomedical tasks. Using data from an intensive care unit and targeting a mortality prediction task, we compare information that has been considered relevant by XAI methods to the opinion of a physician.
Optimizing Multimodal Large Language Models for Detection of Alcohol Advertisements via Adaptive Prompting
Authors: Daniel Cabrera Lozoya; Jiahe Liu; Simon D’Alfonso; Mike Conway
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Adolescents exposed to advertisements promoting addictive substances exhibit a higher likelihood of subsequent substance use. The predominant source for youth exposure to such advertisements is through online content accessed via smartphones. Detecting these advertisements is crucial for establishing and maintaining a safer online environment for young people. In our study, we utilized Multimodal Large Language Models (MLLMs) to identify addictive substance advertisements in digital media. The performance of MLLMs depends on the quality of the prompt used to instruct the model. To optimize our prompts, an adaptive prompt engineering approach was implemented, leveraging a genetic algorithm to refine and enhance the prompts. To evaluate the model’s performance, we augmented the RICO dataset, consisting of Android user interface screenshots, by superimposing alcohol ads onto them. Our results indicate that the MLLM can detect advertisements promoting alcohol with a 0.94 accuracy and a 0.94 F1 score.
Extracting Epilepsy Patient Data with Llama 2
Authors: Ben Holgate; Shichao Fang; Anthony Shek; Matthew McWilliam; Pedro Viana; Joel S. Winston; James T. Teo; Mark P. Richardson
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: We fill a gap in scholarship by applying a generative Large Language Model (LLM) to extract information from clinical free text about the frequency of seizures experienced by people with epilepsy. Seizure frequency is difficult to determine across time from unstructured doctors’ and nurses’ reports of outpatients’ visits that are stored in Electronic Health Records (EHRs) in the United Kingdom’s National Health Service (NHS). We employ Meta’s Llama 2 to mine the EHRs of people with epilepsy and determine, where possible, a person’s seizure frequency at a given point in time. The results demonstrate that the new, powerful generative LLMs may improve outcomes for clinical NLP research in epilepsy and other areas.
How do you know that? Teaching Generative Language Models to Reference Answers to Biomedical Questions
Authors: Bojana Bašaragin; Adela Ljajić; Darija Medvecki; Lorenzo Cassano; Miloš Košprdić; Nikola Milošević
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Large language models (LLMs) have recently become the leading source of answers for users’ questions online. Despite their ability to offer eloquent answers, their accuracy and reliability can pose a significant challenge. This is especially true for sensitive domains such as biomedicine, where there is a higher need for factually correct answers. This paper introduces a biomedical retrieval-augmented generation (RAG) system designed to enhance the reliability of generated responses. The system is based on a fine-tuned LLM for the referenced question-answering, where retrieved relevant abstracts from PubMed are passed to LLM’s context as input through a prompt. Its output is an answer based on PubMed abstracts, where each statement is referenced accordingly, allowing the users to verify the answer. Our retrieval system achieves an absolute improvement of 23% compared to the PubMed search engine. Based on the manual evaluation on a small sample, our fine-tuned LLM component achieves comparable results to GPT-4 Turbo in referencing relevant abstracts. We make the dataset used to fine-tune the models and the fine-tuned models based on Mistral-7B-instruct-v0.1 and v0.2 publicly available.
Low Resource ICD Coding of Hospital Discharge Summaries
Authors: Ashton Williamson; David de Hilster; Amnon Meyers; Nina Hubig; Amy Apon
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Medical coding is the process by which standardized medical codes are assigned to patient health records. This is a complex and challenging task that typically requires an expert human coder to review health records and assign codes from a classification system based on a standard set of rules. Since health records typically consist of a large proportion of free-text documents, this problem has traditionally been approached as a natural language processing (NLP) task. While machine learning-based methods have seen recent popularity on this task, they tend to struggle with codes that are assigned less frequently, for which little or no training data exists. In this work we utilize the open-source NLP programming language, NLP++, to design and build an automated system to assign International Classification of Diseases (ICD) codes to discharge summaries that functions in the absence of labeled training data. We evaluate our system using the MIMIC-III dataset and find that for codes with little training data, our approach achieves competitive performance compared to state-of-the-art machine learning approaches.
Towards ML-supported Triage Prediction in Real-World Emergency Room Scenarios
Authors: Faraz Maschhur; Klaus Netter; Sven Schmeier; Katrin Ostermann; Rimantas Palunis; Tobias Strapatsas; Roland Roller
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: In emergency wards, patients are prioritized by clinical staff according to the urgency of their medical condition. This can be achieved by categorizing patients into different labels of urgency ranging from immediate to not urgent. However, in order to train machine learning models offering support in this regard, there is more than approaching this as a multi-class problem. This work explores the challenges and obstacles of automatic triage using anonymized real-world multi-modal ambulance data in Germany.
Creating Ontology-annotated Corpora from Wikipedia for Medical Named-entity Recognition
Authors: Johann Frei; Frank Kramer
Venue: ACL 2024 (Proceedings of the 23rd Workshop on Biomedical Natural Language Processing)

Abstract: Acquiring annotated corpora for medical NLP is challenging due to legal and privacy constraints and costly annotation efforts, and using annotated public datasets may do not align well to the desired target application in terms of annotation style or language. We investigate the approach of utilizing Wikipedia and WikiData jointly to acquire an unsupervised annotated corpus for named-entity recognition (NER). By controlling the annotation ruleset through WikiData’s ontology, we extract custom-defined annotations and dynamically impute weak annotations by an adaptive loss scaling. Our validation on German medication detection datasets yields competitive results. The entire pipeline only relies on open models and data resources, enabling reproducibility and open sharing of models and corpora. All relevant assets are shared on GitHub.