Do You Have Change? If You Give Me $10, I Will Give You $5 Back

In this issue of The Annals of Thoracic Surgery, Kilic and colleagues1Kilic A. Goyal A. Miller J.K. et al.Performance of a machine learning algorithm in predicting outcomes of aortic valve replacement.Ann Thorac Surg. 2021; 111: 503-510Abstract Full Text Full Text PDF PubMed Scopus (11) Google Scholar evaluated the performance of a machine learning algorithm (extreme gradient boosting, or XGBoost) for predicting seven different postoperative outcomes using data from The Society of Thoracic Surgeons Adult Cardiac Surgery Database (STS ACSD) on patients undergoing isolated surgical aortic valve replacement. The cohort of 243,142 isolated surgical aortic valve replacement patients were split into an 80% training set (to learn and train on the model) and a 20% validation set (to evaluate the model), and this process was iteratively performed 1000 times through bootstrapping. In the end, the authors generate an average XGBoost-based model performance and compare this with the STS ACSD model performance for each of the seven different outcomes, concluding that XGBoost slightly outperformed the STS ACSD models in five of the seven outcomes assessed. The authors should be congratulated on their efforts; however, a few points of clarification are needed in terminology and methodology to understand the findings presented. First, a model is what is derived after you apply an algorithm to a dataset. For example, in the paper by Kilic and colleagues,1Kilic A. Goyal A. Miller J.K. et al.Performance of a machine learning algorithm in predicting outcomes of aortic valve replacement.Ann Thorac Surg. 2021; 111: 503-510Abstract Full Text Full Text PDF PubMed Scopus (11) Google Scholar the dataset used is the cohort of isolated surgical aortic valve replacement patients from the STS ACSD, the algorithm being implemented is XGBoost, and the authors are actually generating 1000 different models (from 1000 different 4:1 splits) with just as many different feature contributions for each of the seven different outcomes being assessed. More simply put, the authors perform cross validation by splitting the entire data set into multiple development and validation sets (1000 to be exact, multiplied by 7). The 20% validation set from one iteration is contaminating the next split and, therefore, there is no truly independent validation set on which to evaluate the XGBoost-based model performance. It is therefore concerning that the slight benefit gained in model performance may be attributed to the degree of information provided by the reintroduced variables. Moreover, by eliminating an independent validation dataset, the authors also eliminate the creation of a useful work product: a clinically useful risk model. Quantification of the amount of informative bias introduced by the contamination of the would-be validation data set due to variable resampling would be illuminating. A different machine learning scheme for risk prediction typically involves splitting the dataset into a development set and a validation set. The development set can then be further split into a training set and a tuning set. The training set will be used to learn the model parameters and the tuning set is used to adjust the hyperparameters (parameters whose value is used to control ‘learning’). This process of splitting the development set into a training and tuning set can be performed as many times as needed to improve model performance. The model generated at the conclusion of this iterative sequence is then tested on a completely independent validation dataset. Machine learning-based risk prediction is influenced by the methodology selected for model development. Understanding the nuances between these different methodologies may even eclipse the final results when one considers the implications of machine learning algorithms. Although we cannot necessarily impugn the authors’ chosen methodology per se, it is important to clearly articulate the findings in light of the limitations introduced by virtue of the authors’ choice. Kilic and colleagues1Kilic A. Goyal A. Miller J.K. et al.Performance of a machine learning algorithm in predicting outcomes of aortic valve replacement.Ann Thorac Surg. 2021; 111: 503-510Abstract Full Text Full Text PDF PubMed Scopus (11) Google Scholar demonstrate that an XGBoost algorithm (using isolated surgical aortic valve replacement data from the STS ACSD) does a good job at predicting the specified outcomes at the cost of producing an immediately available and clinically useful risk prediction model. It is a bit like asking for change for $10 and getting only $5 back. Performance of a Machine Learning Algorithm in Predicting Outcomes of Aortic Valve ReplacementThe Annals of Thoracic SurgeryVol. 111Issue 2PreviewThis study evaluated the performance of a machine learning (ML) algorithm in predicting outcomes of surgical aortic valve replacement (SAVR). Full-Text PDF Seeing Through the FogThe Annals of Thoracic SurgeryVol. 113Issue 3PreviewWe thank Weiss and Karamlou for their commentary1 on our article.2 As with any new technology the increasing adoption of artificial intelligence and machine learning in various forms to cardiothoracic surgery will be incumbent on open discourse and collaboration to best delineate how we as gatekeepers will absorb these innovations. The Society of Thoracic Surgeons has formed a task force to bring clinicians, national database leaders, and artificial intelligence/machine learning experts together to aid in facilitating such collaboration. Full-Text PDF