Stepwise model development and simultaneous validation to keep up with clinically relevant and rapidly changing diagnostic techniques

We have read the letter to the Editor by Heinze et al. [[1]Heinze G. van Smeden M. Wynants L. Steyerberg E. van Calster B. Prediction models: stepwise development and simultaneous validation is a step back.J Clin Epidemiol. 2021; https://doi.org/10.1016/j.jclinepi.2021.07.019Google Scholar] criticising our proposed validation approach of machine learning based prediction models [[2]Haalboom M. Kort S. van der Palen J. Using a stepwise approach to simultaneously develop and validate machine learning based prediction models.J Clin Epidemiol. 2021; https://doi.org/10.1016/j.jclinepi.2021.06.008Google Scholar] with interest. In this letter, we respond to their comments. First we would like to mention that the manuscript has been written as a practical guide for the readership of the Journal of Clinical Epidemiology. We were specifically asked to refrain from an advanced statistical approach. The authors have a valid point with regard to how we used the term external validation, where in some cases split-sample validation should have been used. We have used the proofs to correct this. Another valid point they raised is the issue of using the test cohort during steps one through six instead of only for the final model. We have – after consulting with the editor – changed the table and figure accordingly as well as the text pertaining to this issue. However, we would like to stress that we do not suggest to replace a truly external validation by the validation approach we propose. The situation that we describe in our manuscript is where a new diagnostic test might be highly relevant in clinical practice, but might change over time due to various reasons. We proposed our design as a potential solution to the following situation: (1) a true external validation takes a long time, which will be the case in relatively rare diseases; (2) the modelling techniques used, but also the hardware design can change rapidly, rendering the original model - linked to the original device - irrelevant by the time the new data has been collected during traditional external validation. Moreover, our proposed design provides the opportunity to start using promising emerging less-traditional diagnostic techniques in clinical practice. We agree with the comment that there should be a balance between the number of candidate predictors and sample size. In our manuscript we mention e.g., the use of data compression techniques to obtain much smaller vectors to circumvent this issue. The AUC is proposed in the manuscript as an example when going through steps one to seven, but other diagnostic parameters can of course also be used. In Step eight we do not propose to use the AUC as a single parameter to evaluate the stability of the final 'best' model. As we described in our manuscript, the focus should not only be on the AUC, but also on e.g., sensitivity and specificity. Moreover, this step provides a qualitative impression of whether the model is overfitted or not, to decrease the probability that clinicians will start using a suboptimal and overfitted model in clinical practice, or a model that has not been validated at all! Based on our experience in clinical practice, clinicians will not just replace their previous diagnostic work-up completely, but will first just add the new diagnostic algorithms to evaluate its potential impact.