Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine
Authors: Bergquist Timothy , Schaffter Thomas , Yan Yao , Yu Thomas , Prosser Justin , Gao Jifan , Chen Guanhua , Charzewski Łukasz , Nawalany Zofia , Brugere Ivan , Retkute Renata , Prusokiene Alisa , Prusokas Augustinas , Choi Yonghwa , Lee Sanghoon , Choe Junseok , Lee Inggeol , Kim Sunkyu , Kang Jaewoo , Mooney Sean D , Guinney Justin ,   the Patient Mortality Prediction DREAM Challenge Consortium, Lee Aaron , Salehzadeh-Yazdi Ali , Prusokas Alidivinas , Basu Anand , Belouali Anas , Becker Ann-Kristin , Israel Ariel , Prusokas Augustinas , Winter B , Moreno Carlos Vega , Kurz Christoph , Waltemath Dagmar , Schweinoch Darius , Glaab Enrico , Luo Gang , Chen Guanhua , Zacharias Helena U , Qiao Hezhe , Lee Inggeol , Brugere Ivan , Kang Jaewoo , Gao Jifan , Truthmann Julia , Choe JunSeok , Stephens Kari A , Kaderali Lars , Varshney Lav R , Vollmer Marcus , Pandi Maria-Theodora , Gunn Martin L , Yetisgen Meliha , Nath Neetika , Hammarlund Noah , Müller-Stricker Oliver , Togias Panagiotis , Heagerty Patrick J , Muir Peter , Banda Peter , Retkute Renata , Henkel Ron , Madgi Sagar , Gupta Samir , Lee Sanghoon , Mooney Sean , Kannattikuni Shabeeb , Sarhadi Shamim , Omar Shikhar , Wang Shuo , Ghosh Soumyabrata , Neumann Stefan , Simm Stefan , Madhavan Subha , Kim Sunkyu , Von Yu Thomas , Satagopam Venkata , Pejaver Vikas , Gupta Yachee , Choi Yonghwa , Nawalany Zofia , Charzewski Łukasz , Lee Aaron , Salehzadeh-Yazdi Ali , Prusokas Alidivinas , Basu Anand , Belouali Anas , Becker Ann-Kristin , Israel Ariel , Prusokas Augustinas , Winter B , Moreno Carlos Vega , Kurz Christoph , Waltemath Dagmar , Schweinoch Darius , Glaab Enrico , Luo Gang , Chen Guanhua , Zacharias Helena U , Qiao Hezhe , Lee Inggeol , Brugere Ivan , Kang Jaewoo , Gao Jifan , Truthmann Julia , Choe JunSeok , Stephens Kari A , Kaderali Lars , Varshney Lav R , Vollmer Marcus , Pandi Maria-Theodora , Gunn Martin L , Yetisgen Meliha , Nath Neetika , Hammarlund Noah , Müller-Stricker Oliver , Togias Panagiotis , Heagerty Patrick J , Muir Peter , Banda Peter , Retkute Renata , Henkel Ron , Madgi Sagar , Gupta Samir , Lee Sanghoon , Mooney Sean , Kannattikuni Shabeeb , Sarhadi Shamim , Omar Shikhar , Wang Shuo , Ghosh Soumyabrata , Neumann Stefan , Simm Stefan , Madhavan Subha , Kim Sunkyu , Von Yu Thomas , Satagopam Venkata , Pejaver Vikas , Gupta Yachee , Choi Yonghwa , Nawalany Zofia , Charzewski Łukasz , Lee Aaron , Salehzadeh-Yazdi Ali , Prusokas Alidivinas , Basu Anand , Belouali Anas , Becker Ann-Kristin , Israel Ariel , Prusokas Augustinas , Winter B , Moreno Carlos Vega , Kurz Christoph , Waltemath Dagmar , Schweinoch Darius , Glaab Enrico , Luo Gang , Chen Guanhua , Zacharias Helena U , Qiao Hezhe , Lee Inggeol , Brugere Ivan , Kang Jaewoo , Gao Jifan , Truthmann Julia , Choe JunSeok , Stephens Kari A , Kaderali Lars , Varshney Lav R , Vollmer Marcus , Pandi Maria-Theodora , Gunn Martin L , Yetisgen Meliha , Nath Neetika , Hammarlund Noah , Müller-Stricker Oliver , Togias Panagiotis , Heagerty Patrick J , Muir Peter , Banda Peter , Retkute Renata , Henkel Ron , Madgi Sagar , Gupta Samir , Lee Sanghoon , Mooney Sean , Kannattikuni Shabeeb , Sarhadi Shamim , Omar Shikhar , Wang Shuo , Ghosh Soumyabrata , Neumann Stefan , Simm Stefan , Madhavan Subha , Kim Sunkyu , Von Yu Thomas , Satagopam Venkata , Pejaver Vikas , Gupta Yachee , Choi Yonghwa , Nawalany Zofia , Charzewski Łukasz , Lee Aaron , Salehzadeh-Yazdi Ali , Prusokas Alidivinas , Basu Anand , Belouali Anas , Becker Ann-Kristin , Israel Ariel , Prusokas Augustinas , Winter B , Moreno Carlos Vega , Kurz Christoph , Waltemath Dagmar , Schweinoch Darius , Glaab Enrico , Luo Gang , Chen Guanhua , Zacharias Helena U , Qiao Hezhe , Lee Inggeol , Brugere Ivan , Kang Jaewoo , Gao Jifan , Truthmann Julia , Choe JunSeok , Stephens Kari A , Kaderali Lars , Varshney Lav R , Vollmer Marcus , Pandi Maria-Theodora , Gunn Martin L , Yetisgen Meliha , Nath Neetika , Hammarlund Noah , Müller-Stricker Oliver , Togias Panagiotis , Heagerty Patrick J , Muir Peter , Banda Peter , Retkute Renata , Henkel Ron , Madgi Sagar , Gupta Samir , Lee Sanghoon , Mooney Sean , Kannattikuni Shabeeb , Sarhadi Shamim , Omar Shikhar , Wang Shuo , Ghosh Soumyabrata , Neumann Stefan , Simm Stefan , Madhavan Subha , Kim Sunkyu , Von Yu Thomas , Satagopam Venkata , Pejaver Vikas , Gupta Yachee , Choi Yonghwa , Nawalany Zofia , Charzewski Łukasz , Lee Aaron , Salehzadeh-Yazdi Ali , Prusokas Alidivinas , Basu Anand , Belouali Anas , Becker Ann-Kristin , Israel Ariel , Prusokas Augustinas , Winter B , Moreno Carlos Vega , Kurz Christoph , Waltemath Dagmar , Schweinoch Darius , Glaab Enrico , Luo Gang , Chen Guanhua , Zacharias Helena U , Qiao Hezhe , Lee Inggeol , Brugere Ivan , Kang Jaewoo , Gao Jifan , Truthmann Julia , Choe JunSeok , Stephens Kari A , Kaderali Lars , Varshney Lav R , Vollmer Marcus , Pandi Maria-Theodora , Gunn Martin L , Yetisgen Meliha , Nath Neetika , Hammarlund Noah , Müller-Stricker Oliver , Togias Panagiotis , Heagerty Patrick J , Muir Peter , Banda Peter , Retkute Renata , Henkel Ron , Madgi Sagar , Gupta Samir , Lee Sanghoon , Mooney Sean , Kannattikuni Shabeeb , Sarhadi Shamim , Omar Shikhar , Wang Shuo , Ghosh Soumyabrata , Neumann Stefan , Simm Stefan , Madhavan Subha , Kim Sunkyu , Von Yu Thomas , Satagopam Venkata , Pejaver Vikas , Gupta Yachee , Choi Yonghwa , Nawalany Zofia , Charzewski Łukasz , Lee Aaron , Salehzadeh-Yazdi Ali , Prusokas Alidivinas , Basu Anand , Belouali Anas , Becker Ann-Kristin , Israel Ariel , Prusokas Augustinas , Winter B , Moreno Carlos Vega , Kurz Christoph , Waltemath Dagmar , Schweinoch Darius , Glaab Enrico , Luo Gang , Chen Guanhua , Zacharias Helena U , Qiao Hezhe , Lee Inggeol , Brugere Ivan , Kang Jaewoo , Gao Jifan , Truthmann Julia , Choe JunSeok , Stephens Kari A , Kaderali Lars , Varshney Lav R , Vollmer Marcus , Pandi Maria-Theodora , Gunn Martin L , Yetisgen Meliha , Nath Neetika , Hammarlund Noah , Müller-Stricker Oliver , Togias Panagiotis , Heagerty Patrick J , Muir Peter , Banda Peter , Retkute Renata , Henkel Ron , Madgi Sagar , Gupta Samir , Lee Sanghoon , Mooney Sean , Kannattikuni Shabeeb , Sarhadi Shamim , Omar Shikhar , Wang Shuo , Ghosh Soumyabrata , Neumann Stefan , Simm Stefan , Madhavan Subha , Kim Sunkyu , Von Yu Thomas , Satagopam Venkata , Pejaver Vikas , Gupta Yachee , Choi Yonghwa , Nawalany Zofia , Charzewski Łukasz 
Published: August 2023

Objective

Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question.

Materials and methods

Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system.

Results

The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort.

Discussion

Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data.

Conclusion

This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.