## 1 Introduction  
Machine Learning (ML) models have been a topic of interest in medical diagnostics since the late 20 th century, primarily due to their potential to assist clinicians in detecting diseases and predicting patient outcomes [1, 2, 3, 4, 5, 6]. Over the decades, advancements in computational power and the advent of deep learning architectures, such as convolutional neural networks (CNNs) and transformers [7, 8], have significantly accelerated research in this field [9]. These models promise to transform healthcare, with applications ranging from medical imaging analysis [10, 11], like detecting tumors in X-rays and MRIs [12, 13, 14], to predicting patient outcomes based on electronic health records (EHR) [15, 16, 17, 18, 19]. Recent frameworks have enhanced this capability through temporal learning with dynamic range features and transfer learning techniques that leverage large observational EHR databases [20]. Many recent studies report exceptional accuracy and precision, boasting metrics that often surpass human-level performance in specific diagnostic tasks. For instance, CNN-based models trained on large datasets have demonstrated specificities and accuracies of over 90% in tasks such as breast cancer detection from mammograms [21],or diabetic retinopathy diagnosis from fundus images [22]. These numbers have garnered attention and optimism within the medical community and beyond. AI implementation in medical imaging is demonstrably linked to enhanced efficiency, with roughly two-thirds of studies reporting improvements [23], yet meta-analyses reveal that its benefits vary widely across different applications [24]. Moreover, while many studies note reduced processing times, a significant number also disclose conflicts of interest that could bias study design or outcome estimation [25]. Moreover, these impressive results can be misleading. In many cases, the reported accuracy and precision do not necessarily reflect a model's clinical utility. A closer look often reveals significant methodological issues. A critical issue in the development of ML models for medical diagnostics is overfitting to test sets. This occurs when a model becomes excessively tailored to the specific data it was trained and validated on, leading to inflated performance metrics that do not reflect its ability to generalize to new, unseen data. In many studies, models are evaluated on the same datasets used during development or on overly similar validation sets, which results in a performance boost that can create a false sense of accuracy and precision. This practice, known as data leakage, occurs when there is too much overlap between training and testing data, or when the data is not sufficiently diversified to represent real-world variability. While high accuracy on the training and test datasets may seem promising, it does not guarantee that the model will perform equally well in clinical environments where the data may differ significantly from the training set. Testing the model on entirely unseen datasets, sourced from different hospitals, clinics, or imaging devices, is important for assessing the true generalizability of an ML model. Unfortunately, many studies neglect this critical step, limiting their findings to \"in-domain\" validation, which only tests the model on data from the same or very similar sources. As a result, performance metrics, such as accuracy and precision, may be misleading, failing to capture the true performance of the model in diverse real-world settings. It is also understandable that obtaining relevant external data can be challenging or even impossible due to ethical, legal, or logistical constraints, as well as the inherent limitations in data availability across institutions. In many practical scenarios, researchers are compelled to work with limited external datasets, making comprehensive out-of-domain validation difficult.