A Privacy-preserving Approach to Ingest Knowledge from Proprietary Web-based to Locally Run Models for Medical Progress Note Generation (2024)

Abstract: Clinical documentation is correlated with increasing clinician burden, leading to the rise of automated methods to generate medical notes. Due to the sensitive nature of patient electronic health records (EHRs), locally run models are preferred for a variety of reasons including privacy, bias, and cost. However, most opensource locally run models (including medicalspecific) are much smaller with limited input context size compared to the more powerful closed-source large language models (LLMs) generally available through web APIs (Application Programming Interfaces). In this paper, we propose a framework to harness superior reasoning capabilities and medical knowledge from closed-source online LLMs in a privacypreserving manner and seamlessly incorporate it into locally run models. Specifically, we leverage a web-based model to distill the vast patient information available in EHRs into a clinically relevant subset without sending sensitive patient health information online and use this distilled knowledge to generate progress notes by a locally run model. Our ablation results indicate that the proposed framework improves the performance of the Mixtral model on progress note generation by 4.6 points on ROUGE (a text-matching based metric) and 7.56 points on MEDCON F1 (a metric that measures the clinical concepts overlap).

heading: 1 Introduction
text: Physicians document progress or SOAP (subjective, objective, assessment, and plan) notes in electronic health records (EHRs) periodically to document patient care journey. While abundant patient chart data (e.g., regularly collected lab values) enhances physician assessment of patient progress, it leads to information overload and clinician burden, giving rise to clinician burnout (Tai-Seale et al., 2017), emphasizing the importance of automating this task.
The increasing popularity and capabilities of large language models (LLMs) led to their numer-
ous applications in both general and medical domains (Chen et al., 2024). While the closed-source LLMs available via web APIs (Application Programming Interfaces) generally outperform the locally run alternatives, there is a growing popularity and community support for on-premise models, especially in the medical domain because of several advantages that these models offer such as transparency, adaptability, and information security (Tian et al., 2024). We propose to reap the benefits offered by locally run models while harnessing the strong reasoning capabilities of API-based proprietary LLMs. To this end, there have been numerous efforts toward distilling knowledge from proprietary LLMs (e.g., GPT-4) to train smaller or locally run models (Xu et al., 2024). In the medical domain, most work on such distillation has focused on curating instruction-tuning datasets using superior LLMs for training or tuning smaller models (Wu et al., 2023; Zhang et al., 2023, 2024). Differently, our framework exploits web-based LLMs for achieving a bottleneck task for locally run models formulated in a way that does not spill sensitive patient information to online API-based models.
We formulate the task of progress note generation (PNG) to automatically generate the next note given a patient’s prior progress note and all interim structured chart data (e.g., vital signs). One of the main limitations of the locally run models in tackling PNG is processing and clinically analyzing the vast amount of interim structured chart data (an average of over 1400 rows of tabular data between any pair of subsequent progress notes) – the bottleneck. To overcome this barrier, we leverage an advanced API-based proprietary model to choose clinically relevant structured data rows without sending any real patient information to the online model server. This distilled structured chart information, along with the prior progress note, is used by a locally run model to generate the next progress note.
178

heading: 2 Methods

heading: 2.1 Data
text: We sample the progress notes used in our evaluation from MIMIC-III, a publicly accessible database collected from an intensive care unit (ICU) setting (Johnson et al., 2016). The included pairs of subsequent progress notes were selected if they (1) belong to the same ICU admission and, between their documentation times, there is (2) no other documented progress note and (3) non-empty structured chart data. This resulted in a total of 7089 annotation instances (note pairs) associated with 1616 unique patients and a mean of 1474.9 rows of structured chart data per instance. Due to resource constraints, we randomly sample 100 instances for quantitative evaluation. We additionally perform manual analysis on a sub-sample. The instructions to access the dataset and code used for evaluations are available at GitHub1.
The information in the subjective part of the progress notes is provided by the patient (information more likely to be found in patient-physician conversations) while the objective part is mainly comprised of factual patient data such as laboratory values (oftentimes directly fetched into the note without major modifications). Differently, writing the assessment and plan sections requires a careful examination of the past notes and structured chart data. Thus, in this work, we focus on automatically generating the assessment and plan sections
1github.com/soni-sarvesh/png-privacy-preserving
of a progress note given the previous note and all interim structured chart data.

heading: 2.2 Framework
text: Figure 1 shows the proposed framework’s architecture. The pair of notes in an annotation instance is referred to as Prior and Next notes and the interim structured chart data as Structured Chart Data.

heading: 2.2.1 Data Preparation
text: The Prior note is segmented into different problemspecific sections by (1) identifying clinical problem entities using a clinical concept extraction system, Stanza (Zhang et al., 2021), and (2) applying heuristics over the annotations (e.g., the identified problem entity must be at the beginning of a sentence). Further, we extract the unique available data labels from Structured Chart Data, without the associated clinical data values.

heading: 2.2.2 Proprietary Web-based Model
text: We call the online API-based model once for each problem segment identified from the Data Preparation step. Only the problem entity text span (e.g., Anemia) identified by concept extractor and the unique data labels (e.g., Hemoglobin) without any corresponding values (e.g., 9.6 g/dl) are sent to the web-based model (Figure 2). Multiple structured data elements are collected routinely for subsets of the patients with similar problems. Thus, despite the problem names and data labels coming from a real patient, it is safe to assume that this step does not raise any major privacy concerns, especially in
the absence of any identifiable patient information and the specific data values.
We prompt the model to filter the list of data labels using the supplied problem name such that the resultant labels are useful to document the progress of the patient. The model outputs a list of filtered labels, picking the most important attributes in context of the provided problem name. We chose Anthropic’s Claude 3 Opus (Anthropic, 2024) as our web-based model owing to its superior performance among other proprietary models.

heading: 2.2.3 Interim processing
text: Though the count of filtered data labels for each problem was much smaller, the resultant structured data table with only these labels still contained substantial number of rows. To overcome this, we summarize the rows by aggregating the values associated with data labels based on their data types using simple rules. For numerical values, we reduce the numbers to include only the first and the last measurements with associated chart times along with the mean, minimum, and maximum values. For categorical data, we include the first and the last measurements with chart times along with the most frequent value with its frequency. General corner cases were covered such as reporting the value directly in the case of a single value.

heading: 2.2.4 Locally Run Models
text: The resultant summary from the interim processing step is fed to the locally run model for each problem individually along with the entire problemspecific note text. Additionally, we include the chart times of the Prior (for temporal context) and Next (acting as the note generation time for a fair comparison with ground truth) notes (Figure 3). The model predicts the Next note text for the input problem. We experiment using three locally run models–Biomistral 7B (Labrak et al., 2024), Mixtral 8x7B (Jiang et al., 2024), and LLaMa 2 70B (Touvron et al., 2023). Biomistral is developed by further pre-training the Mistral model (Jiang et al., 2023), an open-weight locally run model, on the PubMed Central Open Access Subset while Mixtral is a mixture-of-experts model based on Mistral. LLaMa 2 is the next generation model from the LLaMa family of LLMs and has shown to outperform the web-based models in some cases.

heading: 2.2.5 Post-processing
text: We combine the generated notes for individual problems to produce a coherent predicted Next note. We use three metrics for our quantitative evaluation– ROUGE (Lin, 2004), BERTScore (Zhang et al., 2019) using RoBERTaLARGE (Liu et al., 2019), and MEDCON (Yim et al., 2023). ROUGE-N calculates N -gram overlap between the predicted and original Next notes while ROUGE-L uses the length of the longest common subsequence and ROUGE-Lsum splits the text into sentences before calculating ROUGE-L. BERTScore measures the cosine similarity between BERT-based contextual embeddings of the tokens in predicted and orig-
inal text. Differently, MEDCON calculates the overlap (using F1-score) between Unified Medical Language System (UMLS) concepts identified in the generated and real notes text.

heading: 3 Results
text: The performance measures in automatically generating progress notes are shown in Table 1. Interestingly, the baseline results from merely returning the same note text as the prior note achieves highest automated evaluation metric scores. Note that this is due to the high textual similarity between the next and previous notes as the progress notes are oftentimes copied forward for editing. The larger models, Mixtral and LLaMa, performed better than Biomistral on the MEDCON metric, while Mixtral performed the best on all three metrics. The ablation results in the sub-sample demonstrate the advantage of our proposed framework that uses problem segments (as opposed to the entire note as input) and distilled structured chart data labels (instead of providing all available data as input). All the models gained improvement in their MEDCON scores with the incorporation of the proposed framework while all the larger models (Mixtral and LLaMa) saw improvements on ROGUE, BERTScore F1 and MEDCON. Of note, Mixtral achieved the largest performance improvements across all the metrics (with as much as 4.6
points on ROGUE-Lsum and 7.56 on MEDCON). Our qualitative analysis of the predictions by the best and worst performing models on 20 instances (Table 2) aligns well with the quantitative results. Further, in our manual evaluation, we found that in most cases the predicted notes contained the relevant interim change information. For instance, “pain and fluid status” in the original next note is appropriately captured in the system prediction by “pain and possible dehydration”. There was minimal evidence of hallucinations (the inclusion of incorrect or irrelevant information in the output) where, in one instance, Biomistral suggested “increasing the dose of vasopressor” while the original note mentioned “off pressors”. Notably, Mixtral did not include incorrect information in the manually evaluated predictions.

heading: 4 Discussion
text: Our results indicate the advantage of tackling the task of PNG by considering individual component problems at a time and leveraging advanced web-based models to transfer knowledge by filtering relevant clinical attributes in structured chart data. Our manual evaluation suggests the predicted notes capture the important updates on patient’s progress. Importantly, Mixtral exhibited capabilities in capturing overall status changes (e.g., sepsis improving), whereas the Biomistral demonstrated
its ability to capture domain knowledge-related updates (e.g., add digoxin 0.25mg daily). Finetuning LLMs leads to specialized domain knowledge (as exhibited by Biomistral), however, it is also shown to reduce general in-context learning abilities (Wang et al., 2023), as seen in Table 1.
Overall, the findings from this paper provide support for the feasibility of the complex task of PNG. Further, it provides a framework for harnessing the reasoning capabilities of proprietary API-based models in a privacy-preserving manner while using a locally run model for handling sensitive patient information.

heading: 5 Limitations
text: The limitations of our framework include its inability to capture new problems that may have emerged in the interval, which is an interesting avenue for future research. Moreover, physicians use information beyond the structured chart data while writing progress notes, e.g., radiology reports. As described earlier, it is challenging to incorporate the interim structured data along with the previous note text in the limited context size of existing onpremise models. Thus, we leave the inclusion of other information sources to future work.

heading: Acknowledgments
text: This work was supported by the intramural research program at the U.S. National Library of Medicine, National Institutes of Health, and utilized the computational resources of the NIH HPC Biowulf cluster (http://hpc.nih.gov).

heading: Future Work
text: The limitations of our framework include its inability to capture new problems that may have emerged in the interval, which is an interesting avenue for future research. Moreover, physicians use information beyond the structured chart data while writing progress notes, e. g. , radiology reports. As described earlier, it is challenging to incorporate the interim structured data along with the previous note text in the limited context size of existing onpremise models. Thus, we leave the inclusion of other information sources to future work