# Generative AI in Medicine and Healthcare: Moving Beyond the Peak of Inflated Expectations'
## 3 Applications and Implications of Generative AI and LLMs in Medicine
### Can We Trust Generative AI? Is It Clinically Safe and Reliable?
#### 3.1.3 Model Explainable Capacity and Performance  
Understanding the explainability and performance of AI models in healthcare is crucial for their reliable and effective use. This subsection delves into the methodologies for visualizing AI model predictions and compares the triage performance of LLMs and untrained doctors.  
A recent study by [PERSON] et al. introduced a novel workflow for understanding the visual signals in medical images that AI classifiers use for predictions [61]. The approach involves several key steps:  
* Train a classifier: Assess whether the medical imagery contains relevant signals for the given task.
* StylEx model: Train a StyleGAN-based (StyleGAN is a generative adversarial network (GAN) first introduced by NVIDIA researchers in December 2018) image generator guided by the classifier to detect and visualize critical visual attributes.
* Attribute visualization: Modify the attributes to create counterfactual visualizations.
* Expert review: Interdisciplinary experts review these visualizations to form hypotheses about the underlying mechanisms.  
This method was applied across eight prediction tasks and three medical imaging modalities (retinal fundus photographs, external eye photographs, and chest radiographs). The study identified attributes capturing clinically known features and unexpected confounders, and new physiologically plausible attributes that could indicate novel research directions. These findings enhance the understanding of AI models, improve model assessments, and aid in designing better datasets. The study concludes by presenting a generative AI-based framework that generates visual explanations for AI predictions in medical imagery, aiming to improve trust in AI models, facilitate novel scientific discoveries, and provide a comprehensive understanding of the visual features influencing AI predictions [61].  
While AI explainability is a highly desirable feature, it is not always possible to achieve due to technology limitations, and AI can still be made useful without it. We should remember in this context that there are many other things, treatments, and therapies we routinely use in healthcare today but do not fully understand or cannot fully explain, with reliable evidence, how exactly they work (mechanism of action), yet we continue using them without waiting for such explanations of how they work to become available before we can successfully harness them for the benefit of our patients [62].  
Unlike explainability, which might not always be available, understanding the performance of a given LLM is essential and doable. For example, a recent study by [PERSON] et al. compared the triage performance of LLMs, specifically ChatGPT, and untrained doctors in emergency medicine. The evaluation used 124 anonymized case vignettes, triaged according to the Manchester Triage System. The results showed that GPT-4-basedChatGPT model and untrained doctors had substantial agreement with professional raters, outperforming GPT-3.5-based ChatGPT. The study concluded that while these LLMs do not yet match professional raters, their performance is comparable to that of untrained doctors. Future improvements are anticipated with technological advancements and specific training of these models [63].