Before revision:  Concerns have motivated international initiatives such as the FDA's led MAQC Consortium , addressing reproducibility of predictive biomarkers by means of appropriate Data Analysis Plans (DAPs) . For instance, repreated cross-validation is a standard procedure meant at mitigating the risk that information from held-out validation data may be used during model selection. We prove here that, many years later, Data Leakage can still be a non-negligible overfitting source in deep learning models for digital pathology. In particular, we evaluate the impact of (i) the presence of multiple images for each subject in histology collections; (ii) the systematic adoption of training over collection of subregions (i.e. "tiles" or "patches") extracted for the same subject.
After revision:  Concerns have motivated international initiatives such as the FDA's led MAQC Consortium , addressing reproducibility of predictive biomarkers by means of appropriate Data Analysis Plans (DAPs) to control for different types of bias, including data leakage from the training to the test set. In the context of digital pathology, the leakage typically lurks in weakly designed experiments not accounting for the subjects in their data partitioning schemes. This issue is then exacerbated when fractions or subregions of slides (i.e. "tiles" or "patches") extracted for the same subject.