# Evaluating the Effectiveness of Large Language Models in Converting Clinical Data to FHIR Format
## 7 Discussion
### Limitations and Future Work  
This study has some limitations. The evaluation was conducted using reports in Spanish from only two clinics (skin care and mammography). This limits the generalizability of the findings to other languages, clinical domains, and report types. Future work should include a more diverse dataset of clinical reports to assess the robustness of the proposed methods across different clinical contexts.  
While we evaluated several LLMs, including the specialized, healthcare-finetuned Llama3-Med42, resource constraints limited our in-depth analysis to this single specialized model. This model proved unsuitable for the FHIR conversion task due to its focus on diagnosis. This highlights the importance of selecting models appropriate for the specific task. Future work should explore LLMs explicitly finetuned for FHIR conversion tasks. Furthermore, a key limitation is the lack of control over the training data used for all LLMs. These models are trained on massive text corpora, making it impossible to know precisely what information they have learned and how this might influence their performance on FHIR conversion.  
The qualitative analysis was conducted by the authors, with GPT-40 used as a complementary tool. While GPT-40 provided additional insights, it exhibited limitations such as occasional hallucinations (identifying non-existent errors) and providing non-substantial feedback, such as \"The hallucination rate was relatively low but could be improved\" or \"The resource mapping could be improved for better accuracy.\" While the overall trends observed by GPT-40 were in line with our own observations, the limitations of LLMs as evaluators for precision, hallucination, and resource mapping should be acknowledged. This aligns with the findings from ref. [41], which suggested evaluating single metrics at a time for more reliable results, although at the cost of increased computational resources.  
Our evaluation primarily focused on the structural validity and information content of the generated FHIR bundles. We did not explicitly evaluate the accuracy of coded values mapped to standard terminologies and ontologies (e.g., LOINC, RxNorm, SNOMED CT)or the use of locally maintained dictionaries and look-up tables within FHIR profiles. This is an important aspect of FHIR compliance that should be addressed in future work.  
Additionally, future research could investigate more advanced prompting techniques, such as prompt chaining or finetuning on FHIR-specific data, to further improve conversion accuracy and reduce the risk of overreliance on examples. Exploring different LLM architectures and sizes could also provide valuable insights. Additionally, developing automated metrics for evaluating FHIR bundle quality would enhance the objectivity and scalability of future studies. Future work should also evaluate the accuracy of mapping clinical concepts to standard terminologies and ontologies within FHIR resources. Finally, future work could investigate the use of FHIR profiles to further constrain and guide the LLMs' output, ensuring adherence to specific implementation requirements.