# Generative AI in Medicine and Healthcare: Moving Beyond the Peak of Inflated Expectations'
## 3 Applications and Implications of Generative AI and LLMs in Medicine
### Can We Trust Generative AI? Is It Clinically Safe and Reliable?
#### 3.1.4 Limitations  
The integration of LLMs in clinical decision-making is not without its limitations. [PERSON] et al. evaluated the performance of a number of Meta Llama 2 LLM derivatives, both generalist (Llama 2 Chat, Open Assistant (OASST), and WizardLM) and medical-domain aligned derivatives (Clinical Camel and MEDITRON--more on these in Section 3.4). They used a curated dataset of 2400 patient cases for their evaluation and identified several key issues [6]:  
* Diagnostic accuracy: LLMs performed worse than clinicians in diagnosing diseases and frequently failed to follow established diagnostic guidelines. This limitation is critical as accurate diagnosis is the cornerstone of effective medical treatment and patient care.
* Interpretation of data: LLMs struggled with interpreting lab results, following instructions accurately, and handling varying quantities and orders of information. This issue is particularly concerning in clinical settings where precise and context-aware interpretation of data is essential.
* Autonomous decision-making: The current capabilities of LLMs are insufficient for autonomous clinical decision-making without extensive human supervision. This limitation suggests that while LLMs can assist clinicians, they are not yet ready to replace human decision-making in critical healthcare environments.
* Integration into clinical workflows: Improvements in fine-tuning and evaluation frameworks are needed to better integrate LLMs into clinical workflows. This includes developing more robust training data, improving model transparency, and ensuring that AI-generated recommendations can be easily understood and validated by human clinicians.  
[PERSON] et al. compared the quality of ChatGPT responses to anesthesia-related medical questions in English and Japanese. They found that English LLM responses were superior in quality to Japanese responses when assessed by bilingual anesthesia experts [64]. Indeed, generative AI models do not process or understand natural language the way humans do [65]. Tokenization, the process of breaking down raw text into smaller units, or tokens, to be processed by an LLM, is a major reason behind some of the strange LLM outputs and limitations observed today including their worse performance in non-English languages. Unless there is a significant breakthrough in tokenization, it appears that new model architectures will be the key to resolving this issue in the future [66].  
In summary, while generative AI and LLMs hold immense potential to transform healthcare through enhanced clinical decision-making, improved diagnostic accuracy, and reduced physician burnout, significant challenges remain. Concerns about bias, variability, and ethical implications necessitate robust regulatory frameworks and comprehensive standards. Explainability and performance of AI models must be enhanced through innovative methodologies, and the limitations of current models need to be addressed through continuous improvements in training and evaluation. The safe and effective integration of these advanced technologies in healthcare requires ongoing research, collaboration among stakeholders, and stringent oversight to ensure they contribute positively to patient care and clinical practice.