Package: healthsageai
HealthSage AI LLM - from clinical note to FHIRIntroductionHealthSage AI's LLM is a fine-tuned version of Meta's Llama 2 13B to create structured information - FHIR Resources - from
unstructured clinical notes - plain text.The model is optimized to process English notes and populate 10 FHIR resource types. For a full description of the scope and limitations, see the performance and limitations header below.Getting startedThis repository consists of the following modules:healthsageai.note_to_fhir.evaluation - compare generated FHIR resources against ground truth/reference resource.healthsageai.note_to_fhir.inference - running inference on the model, either locally or in a containerized environment.Usage examples for both can be found in the jupyter notebooks in the docs folder.The easiest way to get started is to run one of the Jupyter Notebooks on Google Colab and other services, e.g. for inference:Inference: Note to FHIRYou can do Note-to-fhir inference using our NoteToFhir class:fromhealthsageai.note_to_fhir.inferenceimportNoteToFhir13bmodel=NoteToFhir13b()# NoteToFhir8x7b coming soonmodel.translate("Patient John Doe lives in Amsterdam")Evaluation of accuracyOur evaluation tool makes it easy to quantify and inspect the accuracy of generated FHIR resources w.r.t. a ground truth FHIR resource.
For that we designed the FhirScore and FhirDiff models as well as several functions to process and visualize these models.fromhealthsageai.note_to_fhir.evaluation.datamodelsimportFhirScore,FhirDifffromhealthsageai.note_to_fhir.evaluation.utilsimportget_diff,diff_to_dataframefromhealthsageai.note_to_fhir.evaluation.visualsimportshow_difffhir_true,fhir_pred=load_evaluation_dataset()# Placeholder load dataset functiondiff=get_diff(fhir_true,fhir_pred,resource_type="Bundle")show_diff(diff)For a more elaborate walkthrough, seedocs/evaluation.ipynbPublished resourcesThe data sets and models are regularly published toHuggingFace.
The inference API is pushed as a Docker image to Docker Hub.LicensingThe code has been made available under the GNU AGPL 3.0 license. Contact HealthSage AI (hello@healthsage.ai) for commercial licensing options.Contributing & supportContributions are welcome in any form! Please open an issue or a pull request. For support and discussions please visit ourGithub discussionsorSlack community.ValidationThe current version of the Note-to-FHIR model is released as Beta for testing and development purposes only. Its not validated for clinical use.Performance and limitationsScope of the modelThis open sourced Beta model is trained within the following scope:FHIR R410 Resource types:BundlePatientEncounterPractitionerOrganizationImmunizationObservationConditionAllergyIntoleranceProcedure.English languageThe following features are out of scope of the current release:Support for Coding systems such as SNOMED CT and Loinc.FHIR extensions and profilesAny language, resource type or FHIR version not mentioned under "in scope".We are continuously training our model and will make updates available - that address some of these items and more - on a regular basis.Furthermore, please note:No Relative dates:HealthSage AI Note-to-FHIR will not provide accurate FHIR datetime fields based on text that contains relative time information like "today" or "yesterday". Furthermore, relative dates like "Patient John Doe is 50 years old." will not result in an accurate birthdate estimation, since the precise birthday and -month is unknown, and since the LLM is not aware of the current date.Designed as Patient-centric:HealthSage AI Note-to-FHIR is trained on notes describing one patient each.<4k Context window:The training data for this application contained at most 3686 tokens, which is 90% of the context window for Llama-2 (4096)Explicit Null:If a certain FHIR element is not present in the provided text, it is explictely predicted as NULL. Explictely modeling the absence of information reduces the chance of hallucinations.Uses Bundles:For consistency and simplicity, all predicted FHIR resources are Bundled.Conservative estimates:Our model is designed to stick to the information explicitely provided in the text.ID's are local:ID fields and references are local enumerations (1,2,3, etc.). They are not yet tested on referential correctness.Generation design:The model is designed to generate a seperate resource if there is information about that resource in the text beyond what can be described in reference fields of related resources.Test results:Our preliminary results suggest that HealthSage AI Note-to-FHIR is superior to the GPT-4 foundation model within the scope of our application in terms of FHIR Syntax and ability to replicate the original FHIR resources in our test dataset. We are currently analyzing our model on its performance for out-of-distribution data and out-of-scope data.