Is That the Right Dose? Investigating Generative Language Model Performance on Veterinary Prescription Text Analysis (2024)

Abstract: Optimizing antibiotic dosing recommendations is a vital aspect of antimicrobial stewardship (AMS) programs aimed at combating antimicrobial resistance (AMR), a significant public health concern, where inappropriate dosing contributes to the selection of AMR pathogens. A key challenge is the extraction of dosing information, which is embedded in free-text clinical records and necessitates numerical transformations. This paper assesses the utility of Large Language Models (LLMs) in extracting essential prescription attributes such as dose, duration, active ingredient, and indication. We evaluate methods to optimize LLMs on this task against a baseline BERT-based ensemble model. Our findings reveal that LLMs can achieve exceptional accuracy by combining probabilistic predictions with deterministic calculations, enforced through functional prompting, to ensure data types and execute necessary arithmetic. This research demonstrates new prospects for automating aspects of AMS when no training data is available.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 390–397 August 16, 2024. ©2024 Association for Computational Linguistics
390

heading: 1 Introduction
text: Antimicrobial resistance (AMR) has become a major public health concern, as antimicrobials are steadily losing their effectiveness in combating bacterial infections (O’Neill, 2016). AMR is not limited to human medicine; it is also a growing issue among animals (Ekakoro et al., 2022; Cummings et al., 2015), who can acquire and transmit multidrug-resistant pathogens to humans (Guardabassi et al., 2004). Antimicrobial Stewardship (AMS), which has demonstrated effectiveness in improving antimicrobial use in both human and animal healthcare (Davey et al., 2017; Hardefeldt et al., 2022), aims to optimize antimicrobial use to curtail the development and spread of AMR. Accurate dosing is part of this strategy, as overdosing can lead to toxicity and under-dosing can be partic-

heading: Clinical Note Inputs

heading: Consultation Note:

heading: Prescription Label Information:

heading: Inferred Labels for Evaluation
text: ularly perilous as it can select for AMR organisms and lead to poor therapeutic outcomes (Roe et al., 2012; Grill and Maganti, 2011). A pragmatic way to improve dosing accuracy and optimizing antimicrobial use is through decision support systems in clinical settings (Hardefeldt et al., 2018b,a), where targeted dosing recommendations can be made in real-time.
Recent developments in Large Language Models (LLMs) introduce compelling opportunities for automated information extraction and decision support (Bubeck et al., 2023; Nori et al., 2023), as these models obviate the need for extensive labeled data (Brown et al., 2020). Such models can potentially furnish clinicians with data-driven counsel on optimal antimicrobial selection, treatment du-
ration, and dosing intervals, tasks that have been historically reliant on extensive labor-intensive labeled data compilation (Uzuner et al., 2010; Tao et al., 2017). To realize the potential for LLMs for extracting prescription elements, an essential step is empirical assessment of their ability to accurately extract relevant information from clinical text. Given the idiosyncratic nature of LLM training, which leverages instruction tuning rather than conventional training paradigms, it becomes vital to also scrutinize configuration variances for performance optimization (Zheng et al., 2023). Additionally, while the task of extracting elements out of prescriptions was explored in shared tasks such as the 2010 i2b2 challenge (Uzuner et al., 2011), these studies only evaluate the ability to extract text spans without performing numerical conversion. Converting text spans to numerical representations and performing necessary calculations to understand the dose and duration of a given medication are also essential to optimize antimicrobial use. Studies performing such numerical conversions rely on rules-based methods which are notoriously brittle, and only one study we identified made the corresponding algorithm available (Karystianis et al., 2016).
We leverage the VetCompass Australia (McGreevy et al., 2017) corpus, which comprises over 50 million clinical notes from over 200 veterinary clinics across Australia, as our primary data source. Our goal is to extract key information such as the active ingredient, the indication for antimicrobial use, and the dose and duration of the therapy. We assess the performance of LLMs in zero-shot and few-shot learning scenarios for extracting this critical information. By exploring the feasibility of applying LLMs to the VetCompass dataset, we seek to understand their potential in aiding dosing recommendations to support AMS. Specifically:
• We construct a veterinarian-labeled evaluation dataset of 200 clinical notes to study medication dosage extraction from veterinary notes; • Using silver labels generated by a baseline BERTbased ensemble model to provide training examples, we benchmark the performance of LLMs against the baseline model for extracting medication dosage information, their indications, and active ingredients; • While we demonstrate LLMs’ proficiency in element extraction for dose and duration calculations, they falter at arithmetic operations crucial
for deriving these elements (Yuan et al., 2023). We introduce methods to overcome this using functional prompting to combine the probabilistic predictions from the LLMs with deterministic calculations for labelling dosing elements in zero- or few-shot settings.1

heading: 2 Task & Dataset
text: We investigate the task of dose information extraction from veterinary clinical notes. Given textual clinical notes, the task is to extract seven labels, including five entity labels: active ingredient, clinical indication, frequency, medication size, and dosage unit size, along with two derived labels: dose and duration. A sample clinical note, prescription label, and the inferred target labels are illustrated in Figure 1. Prescription label information is provided as an input for all extractions except indication, for reasons of document length. For indication, the model is also given the set of potential indications; for ingredient, the set of potential ingredients. We evaluate accuracy based on exact match between the output label and the ground truth label.
Data Extraction and Label Creation We assemble a subset of 1500 clinical records sourced from VetCompass Australia (McGreevy et al., 2017), focusing on cases where patients received oral antimicrobial treatments as outlined in Hur et al. (2019). To facilitate further calculations, the patient’s weight in kilograms and the total quantity of medication dispensed are also extracted from structured textual fields within the clinical records (Appendix Table 3). We extract the inferred label elements shown in Figure 1 using RxVetBERT, an ensemble model introduced in prior work (Hur et al., 2020, 2022); these inferred labels are used as silver labels for in-context learning examples. The extracted records and labels are partitioned into 1000 records for training, 300 for development, and 200 for test. The test set is reserved exclusively for the final evaluation stage after all prompts have been refined and optimized.
Gold Test Set Annotations To ensure label accuracy in the test set, two expert veterinarians manually annotated the data. Inter-annotator agreement was evaluated using exact match F1 scores. Initial IAA F1 was 0.8 for Indication, 0.985 for Dose, and
1The code and select models used in this study available at https://github.com/havocy28/prescription-text-analyzer
1.0 for Duration, Ingredient, Medication Unit Size and Dose Unit Size. High agreement in dosage and ingredient categories was due to their objectivity. Consensus on indication is more challenging, particularly when multiple clinical events complicated interpretation—e.g., in scenarios involving post-operative complications following traumatic injury, the indication could be correctly interpreted as either the initial injury or subsequent complications. Any annotation discrepancies were resolved through consensus discussion.
Indication and Ingredient Labels Indication labels are based on a subset of Veterinary Nomenclature (VeNOM) codes, a specialized adaptation of SNOMED for veterinary medicine (Brodbelt, 2019). We use the subset of 52 curated by (O’Neill et al., 2019), of which 23 appear in our test set. Ingredient labels are based on unique antimicrobial agents from VetCompass, which consist of 49 unique ingredients, 9 of which occur in our test set.
Dosing Elements Frequency indicates the amount of times per day a dose is given. It must be a numerical value such that it can be used in dosing calculations (e.g., ‘twice daily’ is converted to 2). Dose unit size indicates the amount of medication and must also be converted into a numerical value (e.g., ‘half of a tablet’ is converted to 0.5).
The medication dose and duration can then be calculated using the formulae:
Dose = D ×M
W Duration =
T
F ×D
where D is the Dose Unit Size (number of tablets or volume of liquid), M the Medication Size (tablet size [in mg]), W the Weight of Patient (in kg), T the Total Units Dispensed, and F the Administration Frequency. The dose calculation is designed to tailor the medication dose to the individual’s mass to achieve the optimal therapeutic efficacy while minimizing the risk of toxicity. This is particularly important for veterinary and pediatric patients, where the difference in mass between patients can vary greatly (Waldman et al., 2008).

heading: 3 Methodology
text: We benchmark three LLMs on this task: GPT-3.5 (Brown et al., 2020), GPT-4 (OpenAI, 2023a), and LLAMA2-70B (Touvron et al., 2023), against the baseline ensemble model (RxVetBERT), which combines rule-based methods and VetBERT as described in previous works (Hur et al., 2020, 2022).
Prompt Settings We compare the following: Zero-shot: Utilizes text from the clinical note and/or prescription label, along with the item name, weight of the patient, and the number of units dispensed as input. A prompt for the element being classified is included. No examples are provided.
Few-shot Random Examples: Incorporates randomly-sampled example prescriptions or examination text and inferred labels from the training set as in-context examples (Brown et al., 2020). To manage token limits, we include three labeled examples for prescription prompts and two for indication prompts—the examination text required for the indication prompt were much longer.
Few-shot Similar Examples: Instead of random examples, we use text similarity as a selection criterion to retrieve examples for in-context learning (Zhang et al., 2023; Shi et al., 2023; Lewis et al., 2021). For the retriever, we employ a distilled SBERT model (Wang et al., 2020) to encode text and retrieve examples based on cosine similarity.
Functional Prompting: In the zero-shot setting, we leverage functional prompting (OpenAI, 2023b) with GPT-3.5 and GPT-4 to combine probabilistic outputs with rule-based calculations, enforcing data types for extracted prescription attributes and executing formulaic calculations for Dose and Duration, as detailed in §2. We compare these results with LLAMA2-70B’s configurations for extracting dose unit size and frequency. Additionally, we finetune a VetBERT model (Hur et al., 2020) with silver label data to isolate these attributes, and perform deterministic calculations for dose and duration.
Prompt Tuning To improve the performance of calculating the dose and duration of therapy, we include the formulas for the dose and duration calculations as part of the prompt. The prompts used for evaluation were additionally optimized using the framework proposed by Yang et al. (2023) to iteratively generate a set of prompts using GPT-4, test those prompts on a subset of records from the training set until no improvements were observed after multiple iterations, and keep the prompt with highest accuracy for each element. Final prompts can be found in Appendix A.2.
Postprocessing We remove non-numerical text and retain the first float in the model’s output for enhanced accuracy, except for indication and ingredient which are expected to be strings.

heading: 4 Results and Discussion
text: Overall, our experiments find that LLMs are highly effective at extracting and interpreting numerical elements (e.g., Frequency and Dose Unit Size) necessary to calculate the dose and duration (Table 1). GPT-3.5 and GPT-4 show high accuracy in the zero-shot setting and LLAMA2-70B using incontext learning examples. For all models, integrating probabilistic outputs with deterministic calculations through functional prompting achieves much higher accuracy for dose and duration values compared to directly prompting models for these values (Table 2). Functional prompting provides an effective way to ensure more reliable outcomes in tasks requiring numerical computations when those computations can be explicitly described.
The much smaller finetuned VetBERT achieves modest results, similar to silver label accuracy for dose unit size and frequency of administration,
which suggests it could achieve higher accuracy with more accurate training labels. For active ingredient, we find LLMs to be highly effective with appropriate prompt tuning. Error analysis of initial results found that there were many errors for multi-ingredient medications, e.g., Amoxycillin Clavulanate incorrectly identified as Amoxycillin. Through prompt tuning, we identified the most effective way to overcome this as including the following prompt text: “focus on the active ingredients and note them all if they were present.”
Nonetheless, accurately pinpointing the primary indication for antimicrobial administration continues to be a challenging task, as evidenced by the lower inter-annotator agreement score for indications as discussed in §2, and the relatively poor performance of LLMs on this subtask. A closer examination of the errors reveals that they largely occurred in instances where the indication is ambiguous, similar to the complications noted earlier. Refining the labeling schema for indications is a promising avenue for mitigating this issue.
Conclusion This paper provides a framework for LLMs to extract essential prescription data from veterinary text, such as dose, duration, and active ingredients for supporting AMS efforts. We overcome limitations in calculating elements such as dose and duration by integrating probabilistic outputs with deterministic calculations through functional prompting, even in zero-shot settings. Future work should consider evaluation for human clinical applications, given the potential contributions of this approach to broader healthcare.

heading: Limitations
text: The efficacy of in-context learning for models in the few-shot similar setting may be constrained by the precision of RxVetBERT, which was employed to furnish the examples used in the prompts. Random sampling was used to create a test set mirroring the full dataset population; this limited the diversity of specific disease syndromes in the test data, and may not provide a complete assessment of the models’ capabilities.
LLMs are still prone to errors, even though they demonstrate high performance on evaluation datasets. When using an LLM for clinical decisions, it is critical that the final decisions involve clinicians as LLMs in their current state may still fall short. While our framework excels in identifying active ingredients, it faces challenges in ascertaining exact indications for medication, a more subjective task, signaling a potential direction for future work.

heading: Acknowledgements
text: This work was supported in part by the National Institutes of Health, National Library of Medicine (NLM) University of Washington Biomedical Informatics and Data Science Research Training Program (Grant Nr LM 007442). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This research was undertaken with the assistance of information and other resources from the VetCompass Australia consortium under the project “VetCompass Australia: Big Data and Real-time Surveillance for Veterinary Science”, which is supported by the Australian Government through the Australian Research Council LIEF scheme (LE160100026).

heading: A Appendix

heading: A.1 Prompt Design Details
text: This section details the prompt template used and provides an example prompt. Each initial prompt was generated using GPT-4 by prompting the model to generate 10 additional prompts for accomplishing the task. Each task was tested using GPT-3.5 and the process was repeated until the additional prompts were no longer improving the performance after 3 successive iterations. For dose and duration, which also required the steps for performing arithmetic, the formula for the necessary arithmetic was provided.
The prompt templates were designed as follows: {PROMPT}
{Examples} (omitted in zero-shot settings)
{Instance to label}
Here is an example prompt for dose under the few-shot random setting:
Output the dosage in mg/kg. Dose is determined by multiplying the total dose units given per administration multiplied by the size of the medication in mg, dividing by the weight of the patient in kg to determine the mg per kg
** Example: ** Item Label: Disp By: ***: Dog 21.00 x Clinacin
Tabs 150Mg One \& half (1.5) tablets twice a day with food ***
** Item Name: Clinacin Tabs 150Mg (100) Clindamycin ** Weight: 30kgs ** Medication Unit Size: 150.0
** Units Dispensed: 21.0 ** Dose: 7.5
** Example: ** Item Label: PM:Disp:***: Cat 7.00 x Baytril 50Mg
Tab Half (1/2) tablet once a day Give until finished. ***
** Item Name: Baytril 50Mg Tab (100) (enrofloxacin) ** Weight: 5kgs ** Medication Unit Size: 50.0 ** Units Dispensed: 7.0 ** Dose: 5
** Example: ** Item Label: Vet: ***: *** : Dog 10.00 x Veraflox
Dog 60mg Give ONE (1) tablet ONCE a day Give until finished; ***
** Item Name: Veraflox Dog 60mg (70) (pradofloxacin) ** Weight: 30kgs ** Medication Unit Size: 60.0 ** Units Dispensed: 10.0 ** Dose: 2
** Instance to Label: ** Item Label: Vet: ***: Dog 10.00 x Clavulox Tabs
500Mg One (1) tablet twice a day Give until finished. with food ***
** Item Name: Clavulox (Clavulanic Acid) Tabs 500Mg (100) **\\ Weight: 29kgs ** Medication Unit Size: 500.0 ** Units Dispensed: 10.0 ** Dose:

heading: A.2 Prompts Used
text: This section details the example prompts used for each label in the task.
• Active Ingredient: “Referencing the trade name, choose the active ingredient from the Ingredients List that forms the medication. For combination drugs, ensure to select the ingredient with all components.”
• Clinical Indication: “Using the provided list of possible indications, give the most likely indication for the antimicrobial administration. If unclear from the text, label as unknown.”
• Frequency: “How many times per day is the medication given?”
• Medication Size: “What is the medication unit size in mg?”
• Dosage Unit Size: “How many units of the medication are given per dose?”
• Overall Dose: “Output the dosage in mg/kg. Dose is determined by multiplying the total dose units given per administration multiplied by the size of the tablet in mg, dividing by the
weight of the patient in kg to determine the mg per kg.”
• Treatment Duration: “Calculate the length of administration (in days) for the given prescription. To determine the length of administration, find the total number of tablets or doses dispensed and divide by the number of doses given per day.”

heading: A.3 Additional Evaluations
text: While evaluations were performed on all aspects of the prescription text, we omitted the performance of the elements which could be extracted directly out of the text, which were required for the dose calculations but did not require any conversion into numerical values, this included the Weight, Total Units, or Medication Size. We have included this in Table 3.

heading: Future Work
text: While our framework excels in identifying active ingredients, it faces challenges in ascertaining exact indications for medication, a more subjective task, signaling a potential direction for future work