# Evaluating the Effectiveness of Large Language Models in Converting Clinical Data to FHIR Format
## 5 Methodology  
The workflow for converting unstructured clinical reports into FHIR bundles and evaluating their quality is summarized in Figure 2. Through this process, we aimed to enhance healthcare interoperability by providing insight into the best model, prompt strategies, and parameters to convert unstructured clinical reports to FHIR. Evaluating the quality of the conversions generated by LLMs is a challenging task, especially in the sensitive domain of healthcare [38]. The evaluation of long-form generation from LLMs remains a nascent area of work [39]. In this study, we evaluated the quality of the conversion through three methods. First, we ensured that the resulted data adhered to FHIR standards by implementing data type conversions and value transformations for elements requiring specific data types (e.g., Boolean, integer, string, decimal). We call this metric the Failure Rate, which corresponds to the proportion of FHIR bundles that respect FHIR standards using the HAPI FHIR validation tool. A lower failure rate indicates better adherence to FHIR standards. Second, we developed an approach to automatic evaluation using a multifaceted, iterative design methodology. We leveraged the capabilities of the LLMs themselves to assess the quality of the conversion. Specifically, we used GPT-40 ([[https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com](https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com)]([https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com](https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com)) accessed on 17 February 2025) [40], an advanced external commercial LLM, as the Evaluator. We chose GPT-4o as the evaluator for several reasons. First, GPT-4o is known for its advanced capabilities in understanding and generating coherent, contextually relevant text, making it well suited for evaluating the quality of the converted outputs. Second, its extensive training on a diverse range of tasks and data ensures that it can provide robust and reliable evaluations across various metrics. Third, using an external evaluator helps to mitigate potential biases that might arise from using the same model for both conversion and evaluation. This model evaluates the conversion performed by other LLMs, which we refer to as Convertor LLMs. We designed the evaluation by instructing the Evaluator to compare the model answers with the original clinical document to determine if the Convertor LLMs showed high capacities across our three metrics: (a) precision, (b) hallucination, and (c) resource mapping accuracy:  
Figure 2: This diagram illustrates the process of converting clinical reports into FHIR bundles using LLMs. The pipeline includes the clinical report input, conversion to FHIR using LLMs with various prompting techniques, including zero-shot and few-shot prompting, as well as basic, two-step, and chain-of- thought methods. The output is validated using HAPI FHIR to ensure it is a correct FHIR bundle. Finally, an external LLM evaluates the quality of the FHIR bundle by comparing it with the original clinical report. Precision: Does the converted document correctly identified the information described in the clinical case? High precision indicates accurate identification and representation of clinical information. 2. Hallucination: Does the converted document include information that is not described in the clinical case? A low hallucination rate indicates that the conversion accurately reflects the original clinical report without introducing extraneous information. 3. Resource Mapping Accuracy: Are data elements correctly mapped to their appropriate FHIR resources? High resource mapping accuracy ensures that all relevant data elements are correctly represented in the FHIR bundle. To perform the automatic evaluation, we used a set of pre-defined prompts to query the Evaluator and elicit assessments of the converted document (refer to Appendix B.2 for detailed prompts). In line with [PERSON] et al. [41], we adjusted the temperature of the Evaluator to 0.