# Evaluating the Effectiveness of Large Language Models in Converting Clinical Data to FHIR Format
## 6 Results
### Hallucination Rate  
We depict in Table 4 the mean and standard deviation of the hallucination rate for each model's conversion from unstructured data to FHIR. The table highlights the effectiveness of different prompting techniques and temperature levels in mitigating hallucinations.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & & & \multicolumn{4}{c}{**Precision**} \\ \hline \multirow{3}{*}{**Clinic**} & \multirow{3}{*}{**Model Name**} & \multirow{3}{*}{**Prompt Tech.**} & \multicolumn{2}{c}{**Zero-Shot**} & \multicolumn{2}{c}{**Few-Shot**} \\ \cline{3-8}  & & & **Low** & **High** & **Low** & **High** \\ \hline \multirow{8}{*}{Skin Care} & \multirow{3}{*}{Gemini} & Basic & 2.09 (1.87) & 1.73 (1.62) & 4.45 (3.33) & 6.45 (2.73) \\ \cline{3-8}  & & \multirow{3}{*}{Gemini} & Two-Step & 7.64 (0.5) & 7.64 (0.5) & 7.82 (0.4) & 7.45 (0.52) \\ \cline{3-8}  & & & CoT & 5.18 (2.96) & 5.27 (2.94) & 6.27 (2.65) & 7.64 (0.5) \\ \cline{3-8}  & & \multirow{3}{*}{Mistral} & Basic & \(\times\) & 1.82 (2.71) & 5.56 (3.2) & 7.64 (0.5) \\ \cline{3-8}  & & & \multirow{3}{*}{Two-Step} & 7.18 (0.4) & 6.7 (2.06) & 5.06 (3.11) & 7.44 (0.53) \\ \cline{3-8}  & & & CoT & 3.91 (3.36) & 3.27 (3.17) & 7.55 (0.52) & 7.55 (0.52) \\ \hline \multirow{8}{*}{Mammography} & \multirow{3}{*}{Gemini} & Basic & 1.44 (1.33) & 1.56 (1.33) & 7.78 (0.97) & 7.44 (0.53) \\ \cline{3-8}  & & & \multirow{3}{*}{Two-Step} & 7.78 (0.44) & 7.67 (0.5) & 7.67 (0.5) & 7.89 (0.33) \\ \cline{3-8}  & & & CoT & 3.78 (3.42) & 4.33 (3.28) & 8.0 (0.87) & 7.67 (0.5) \\ \cline{3-8}  & & & Basic & \(\times\) & \(\times\) & 5.86 (3.21) & 7.44 (0.53) \\ \cline{3-8}  & & & Two-Step & 7.89 (0.93) & 7.44 (0.53) & 6.0 (3.35) & 7.56 (0.53) \\ \cline{3-8}  & & & CoT & 4.78 (3.6) & 4.9 (3.38) & 7.62 (0.52) & 7.89 (0.33) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Mean and standard deviation of precision scores for different clinics, models, prompt techniques, and example presence at various temperatures. Color coding indicates value ranges: 1–2.5 (red), 2.5–5 (orange), 5–7.5 (olive green), and 7.5–10 (green). Standard deviations are shown in parentheses. Higher scores are favorable while \(\times\) denotes no valid conversion.  
Key observations include consistently lowest hallucination rates with two-step conversion across all models and prompting techniques. By first structuring the information and then converting it to FHIR, the model can better understand the context and relevance of the data, leading to more accurate and less hallucinogenic outputs. Then, providing examples in the prompt slightly increases the hallucination rate compared to zero-shot prompting. This suggests that while examples can guide the model, they may also introduce biases or irrelevant information, leading to higher hallucination rates. While higher temperatures generally increase the hallucination rate, especially with zero-shot prompting, we do not observe difference in these experiments.  
Finally, Mistral tends to have slightly higher hallucination rates compared to Gemini with zero-shot prompting, while the opposite is observed with few-shot prompting.