# Evaluating the Effectiveness of Large Language Models in Converting Clinical Data to FHIR Format
## 6 Results
### Qualitative Analysis
#### 6.6.1 Impact of Prompt Techniques  
With basic prompting, the LLMs produce structured reports containing important information. However, these reports do not follow FHIR standards. For instance, they often use non-standard terms (e.g., 'Doctor' instead of 'Practitioner' or 'Study' instead of 'Observation'), making automatic conversion to FHIR impossible. The basic approach, while straightforward, lacks the necessary guidance to ensure compliance with FHIR standards, resulting in reports that are informative but not structurally valid.  
Using chain-of-thought (CoT) prompting, we found that the models accurately identify the relevant information from the source documents. Implementing a second CoT round, intended to allow the models to correct any initial errors, increases the number of valid FHIR bundles generated. However, this approach does not consistently produce  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Clinic**} & \multirow{2}{*}{**Model Name**} & \multirow{2}{*}{**Prompt Tech.**} & \multicolumn{3}{c}{**Zero-Shot**} & \multicolumn{2}{c}{**Few-Shot**} \\ \cline{3-8}  & & & **Low** & **High** & **Low** & **High** \\ \hline \multirow{8}{*}{Skin Care} & \multirow{4}{*}{Gemini} & Basic & 1.91 (1.87) & 1.55 (1.29) & 4.73 (3.58) & 6.09 (2.59) \\ \cline{3-8}  & & Two-Step & 7.36 (0.5) & 7.27 (0.65) & 7.55 (0.82) & 7.55 (0.52) \\ \cline{3-8}  & & CoT & 4.91 (2.98) & 5.09 (2.95) & 6.27 (2.69) & 7.18 (1.08) \\ \cline{2-8}  & \multirow{4}{*}{Mistral} & Basic & \(\times\) & 1.82 (2.71) & 5.25 (3.0) & 7.27 (0.65) \\ \cline{3-8}  & & Two-Step & 7.82 (0.4) & 7.0 (2.16) & 5.59 (3.52) & 7.44 (0.73) \\ \cline{3-8}  & & CoT & 3.73 (3.23) & 3.36 (3.32) & 7.73 (0.9) & 7.64 (0.67) \\ \hline \multirow{8}{*}{Mammography} & \multirow{4}{*}{Gemini} & Basic & 1.44 (1.33) & 1.44 (1.33) & 6.78 (1.64) & 6.89 (1.45) \\ \cline{3-8}  & & Two-Step & 7.67 (0.87) & 7.33 (1.0) & 7.67 (1.0) & 7.44 (1.01) \\ \cline{1-1} \cline{2-8}  & & CoT & 3.56 (3.5) & 4.22 (3.49) & 7.44 (1.13) & 7.33 (0.5) \\ \cline{1-1} \cline{2-8}  & \multirow{4}{*}{Mistral} & Basic & \(\times\) & \(\times\) & 5.71 (3.17) & 7.44 (1.01) \\ \cline{1-1} \cline{2-8}  & & Two-Step & 8.11 (1.05) & 7.89 (0.78) & 6.07 (3.45) & 7.33 (0.71) \\ \cline{1-1} \cline{2-8}  & & CoT & 4.44 (3.28) & 4.9 (3.38) & 7.25 (0.71) & 7.89 (1.17) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average and standard deviation of resource mapping accuracy for different clinics, models, prompt techniques, and example presence at various temperatures. Color coding represents value ranges: 1–2.5 (red), 2.5–5 (orange), 5–7.5 (olive green), and 7.5–10 (green). Standard deviations are shown in parentheses. Lower values are desirable and \(\times\) represents no successful conversion.  
fully valid bundles and frequently results in a reduction in the number of information elements included.  
The two-step conversion approach performed well. This method involves first structuring the information and then converting it to FHIR. It correctly identifies the information and successfully converts it to FHIR, demonstrating a more robust and reliable method for this task. The two-step approach ensures that the information is first organized in a way that the models can understand and is then accurately mapped to FHIR resources, resulting in more complete and valid bundles.