Leveraging Large Language Models (LLMs) to create a chatbot assistant for the retrieval of medication information
Creators: name: Tam, Bernard
name: Chng, Sue Inn
name: Quiroz Aguilera, Juan
Publisher: name: UNSW Sydney
Year: 2024

description: Aim: Develop a chatbot powered by a large language model (LLM) that provides personalized information on medication side effects and contraindications. Background: Medication side effects and contraindications play a significant role in medication non-adherence, which leads to higher medical expenses and worse patient outcomes. Dr. Google, WebMD, online forums, and mobile apps offer information on drug interactions and adverse reactions, but the accuracy and coverage of these tools vary greatly. Users struggle with discerning appropriate information and finding reliable sources of information, especially when taking multiple medications and dealing with multiple chronic conditions. LLMs like ChatGPT provide a new source of medical information for patients, but LLMs are known to hallucinate convincing, but factually incorrect responses. Methods: We developed a chatbot with an LLM and retrieval augmented generation (RAG). The chatbot uses trusted external data sources — the United States Food and Drug Administration (FDA) and DrugBank — to collect information on medications based on the user query. The chatbot is made up of agents that interact with GPT-4 to create responses to user queries. Subtasks handled by task-specific agents include: obtaining drug adverse reactions from trusted sources, prompting the user for clarification or additional information, retrieving a user’s past conversations with the chatbot from a database, and generating a final response using all collected information. The chatbot is accessible via SMS (implemented with Twilio), providing a minimalist interface that removes usability barriers for users. Conversations with the chatbot are stored in a database, allowing future responses to be personalized based on a user’s medication history and concerns. Results: We designed our chatbot with SMS messaging as the main interface to mimic the way a patient might interact with a health professional or pharmacist over text messages. SMS also empowers individuals’ from rural and underserved communities access to our chatbot, since it does not require an internet connection. For a preliminary evaluation, we compared our chatbot against ChatGPT and Llama 2. We selected 20 common drug-drug interactions in patients with cardiovascular disease. Our chatbot identified the correct occurrence of a contraindication in all 20 test cases, whereas Llama 2 identified 19 out of 20 and ChatGPT identified 12 out of 20. Our chatbot also identified the consequence of the drug interactions for 18 of the 20 test cases, whereas ChatGPT and Llama 2 only identified consequences for 11 and 14 of the 20 test cases respectively. Our chatbot has been released as open source, available on GitHub (https://github.com/Yakultie/drug-interactions-sms-chatbot). The chatbot provides a template for the research community to develop the next generation of medication adherence tools that can leverage the power of LLMs. Our framework allows for the customization of medical sources (depending on the target application), customizing responses for a particular condition (cardiovascular disease or HIV), or choice of LLM (Llama 2 instead of GPT-4, depending on privacy or security concerns).
description: Conference on Health IT and Analytics; Washington, DC; 2024-05-03 - 2024-07-04