# Generative AI in Medicine and Healthcare: Moving Beyond the Peak of Inflated Expectations'
## 3 Applications and Implications of Generative AI and LLMs in Medicine
### Can We Trust Generative AI? Is It Clinically Safe and Reliable?
#### 3.1.1 Regulation, Regulatory Frameworks, and Ethics  
[PERSON] et al. underscored the potential of LLM-generated summaries to streamline clinical tasks and reduce physician burnout. However, significant concerns were raised regarding their variability, potential for bias, and the lack of US FDA oversight. The authors emphasized the necessity for comprehensive standards, rigorous testing, and regulatory clarifications to ensure the safe and effective use of LLMs in clinical settings. It was pointed out that LLMs can exhibit sycophancy bias, tailoring responses to perceived user expectations, which may lead to confirmation biases in clinical decision-making. Additionally, small errors in summaries, such as adding words not present in the original data, can significantly impact clinical outcomes. The authors advocated for comprehensive standards and careful prompt engineering to mitigate these issues and ensure the safe use of LLMs in clinical environments [35]. [PERSON] et al. highlighted the critical importance of ethical considerations and robust regulatory frameworks for the safe and effective implementation of AI in public health. It called for comprehensive guidelines to address issues such as data privacy, bias, and accountability in AI-driven healthcare solutions. Moreover, it stressed the need for collaboration between policymakers, healthcare professionals, and AI developers to create regulations that ensure transparency and public trust in AI technologies [36]. The regulatory landscape for AI as a medical device (AlaMD) [37] is evolving, with some voices questioning the suitability of current regulatory frameworks. Google MedLM [38], for example, poses unique regulatory challenges (for any clinical application based on it) as it may be classified as software of unknown provenance (SoUP), complicating its path to approval [39]. Future clinical solutions incorporating Google MedLM and similar developments will need to meet stringent regulatory requirements set by agencies worldwide, potentially necessitating new or updated regulatory paradigms [40]. These concerns are echoed in recent discussions about the need to update regulatory frameworks that were originally optimized for earlier generations of AI. [PERSON] et al. describe three epochs of AI with fundamentally different capabilities and risks: AI 1.0 or rule-based/symbolic AI, AI 2.0 or deep learning models, and AI 3.0 covering foundation models and generative AI [41]. As of 7 August 2024, the US FDA has approved 950 SaMDs that fall under AI 1.0/AI 2.0 but none that belong to AI 3.0 [42]. [PERSON] and [PERSON] argue that the current approaches to assuring AI safety and efficacy are optimized for older forms of AI and that the regulation of clinical generative AI may require the development of new regulatory paradigms that can specifically address the complexities of LLMs [40]. It is noteworthy that the two articles by [PERSON] and [PERSON] [40] and [PERSON] et al. [41], whilst representing important contributions to the ongoing debate on the subject of medical generative AI regulation, feature Google employees among their authors. Caution should be exercised against potential conflicts of interest, especially when influential companies like Google are involved in shaping new regulation. This is not to say that these companies should not be involved in formulating new regulation. They should be included as a key (but not sole) stakeholder, and their contribution is vital, as without these companies and their very costly generative AI infrastructure and research, we would not have gotten to where we are today (e.g., [43]). However, it should always be remembered that current regulation focuses on claims, efficacy, and safety--principles that do not and should never change across AI generations. Any new clinical generative AI regulation should not attempt to \"dilute\" existing quality and safety principles and standards in any way, but should rather evolve to maintain or improve upon the high standards that SaMD/AIaMD regulation adheres to today.