## 4 Discussion
### Highlighted Impressions and Overall Themes
#### 4.1.3 Ethical and Explainability Concerns in AI Applications  
The integration of AI into clinical settings brings forth a host of challenges and ethical considerations. As AI models seek to influence clinical decision-making, concerns regardingbias, errors, and the interpretability of these systems become top of mind. AI models, while powerful, can inadvertently perpetuate biases present in their training data, leading to skewed or unfair medical decisions. Errors in AI predictions, especially in high-stakes medical scenarios, can have suboptimal consequences, raising questions about liability and oversight. In cases of medical errors or misdiagnoses involving AI, determining the responsible party, whether it is the healthcare provider, the AI developer, or another entity, is a complex and unsolved problem. This complexity underscores the necessity for clear guidelines and regulations around the use of AI in healthcare.  
One major concern with deep learning techniques being applied in the clinic is their \"black box\" nature, or the fact that the architecture of these models does not allow for easy interpretation. Unlike a regression model, in which the importance of coefficients can be more easily weighed and measured, deep learning models are much more complex, involving numerous nodes, edges, layers, and weights, making insight into how the input affects the output a bit more complicated. One paper in this list argues that explainable AI is crucial for optimal healthcare delivery and provides a survey of interpretability approaches that can be used in various application settings [29]. This paper posits that transparency, in the form of explainability and interpretability, measures reliability. The other work included in this list provides a different point of view. The authors argue that, as of now, incorporating transparency techniques into deep learning models is a \"false hope\" and cannot reliably mitigate bias in their applications [31]. They showcase how various failure cases can still obfuscate the clinical decision-making process, such as explainability techniques being used for justification of a decision, reducing vigilance, and hampering practitioners' ability to detect when a serious error was made for an individual patient. Finally, they argue that more comprehensive testing and external validation can be used to mitigate biases. These studies both agree that there is a need for more advanced techniques that offer greater and more accurate transparency.  
The other subtopic of this category revolves around the need for regulation of AI-based medical devices. One work highlights the challenges faced by regulatory bodies in keeping pace with the rapid development of AI technologies in healthcare [30]. The authors identified 222 and 240 devices approved in the USA and Europe, respectively. The paper underscores the need for stringent regulations to ensure the efficacy, safety, and quality of AI/ML-based medical devices. Governmental regulation is particularly important given the rising number of such devices being approved for clinical use. The paper calls for more transparency in the regulatory process to bolster public trust in these advanced technologies. Put together, this topic explores the ethical and explainability issues that need to be addressed in AI for healthcare. These issues include the potential for bias and errors, the need for better transparent and interpretable AI models, and the development of robust regulatory frameworks. Addressing these concerns will be necessary for long-term success in the responsible and effective use of AI in improving patient care and outcomes.