Effectiveness, Explainability and Reliability of Machine Meta-Learning Methods for Predicting Mortality in Patients with COVID-19: Results of the Brazilian COVID-19 Registry
Authors: Miranda de Paiva Bruno Barbosa , Delfino-Pereira Polianna , de Andrade Claudio Moisés Valiense , Gomes Virginia Mara Reis , Lima Maria Clara Pontello Barbosa , Souza-Silva Maira Viana Rego , Carneiro Marcelo , Martins Karina Paula Medeiros Prado , Sales Thaís Lorenna Souza , de Carvalho Rafael Lima Rodrigues , Pires Magda C. , F. Ramos Lucas Emanuel , Silva Rafael T. , Bezerra Adriana Falangola Benjamin , Schwarzbold Alexandre Vargas , Nunes Aline Gabrielle Sousa , Maurílio Amanda de Oliveira , Scotton Ana Luiza Bahia Alves , Costa André Soares de Moura , Castro Andriele Abreu , Farace Bárbara Lopes , Cimini Christiane Corrêa Rodrigues , De Carvalho Cíntia Alcantara , Silveira Daniel Vitório , Ponce Daniela , Pereira Elayne Crestani , Manenti Euler Roberto Fernandes , Cenci Evelin Paola de Almeida , Lucas Fernanda Barbosa , Rodrigues Fernanda D’Athayde , Anschau Fernando , Botoni Fernando Antonio , Graça Aranha Fernando , Bartolazzi Frederico , Bastos Gisele Alsina Nader , Vietta Giovanna Grunewald , Nascimento Guilherme Fagundes , Noal Helena Carolina , Duani Helena , Vianna Heloisa Reniers , Guimarães Henrique Cerqueira , Gomes Isabela Moraes , Salles Martins Costa Jamille Hemétrio , da Fonseca Jéssica Rayane Corrêa Silva , Guimarães Júlia Di Sabatino Santos , de Morais Júlia Drumond Parreiras , Rugolo Juliana Machado , Batista Joanna D’arc Lyra , de Alvarenga Joice Coutinho , Chatkin José Miguel , Ruschel Karen Brasil , Moreira Leila Beltrami , de Oliveira Leonardo Seixas , Zandoná Liege Barella , Pinheiro Lílian Santos , Monteiro Luanna da Silva , Sousa Lucas de Deus , Kopittke Luciane , Viana Luciano de Souza , de Castro Luis César , Assis Luisa Argolo , Santos Luisa Elem Almeid , Cabral Máderson Alvares de Souza , Raposo Magda Cesar , Floriani Maiara Anschau , Ferreira Maria Angélica Pires , Bicalho Maria Aparecida Camargos , de Godoy Mariana Frizzo , Nogueira Matheus Carvalho Alves , de Figueiredo Meire Pereira , Guimarães-Júnior Milton Henriques , De Sordi Mônica Aparecida de Paula , Sampaio Natália da Cunha Severino , de Oliveira Neimy Ramos , Assaf Pedro Ledic , Lutkmeier Raquel , Valacio Reginaldo Aparecido , Finger Renan Goulart , Senger Roberta , Menezes Rochele Mosmann , Silva Rufino de Freitas , Francisco Saionara Cristina , Guimarães Silvana Mangeon Mereilles , Araújo Silvia Ferreira , Oliveira Talita Fischer , Kurtz Tatiana , Fereguetti Tatiani Oliveira , de Oliveira Thainara Conceição , Diniz Thulio Henrique Oliveira , Ribeiro Yara Cristina Neves Marques Barbosa , Ramires Yuri Carlotto , Gonçalves Marcos André , Marcolino Milena Soriano 
Published: November 2021

Objective

To provide a thorough comparative study among state-of-the-art machine learning methods and statistical methods for determining in-hospital mortality in COVID-19 patients using data upon hospital admission; to study the reliability of the predictions of the most effective methods by correlating the probability of the outcome and the accuracy of the methods; to investigate how explainable are the predictions produced by the most effective methods.

Materials and Methods

De-identified data were obtained from COVID-19 positive patients in 36 participating hospitals, from March 1 to September 30, 2020. Demographic, comorbidity, clinical presentation and laboratory data were used as training data to develop COVID-19 mortality prediction models. Multiple machine learning and traditional statistics models were trained on this prediction task using a folded cross-validation procedure, from which we assessed performance and interpretability metrics.

Results

The Stacking of machine learning models improved over the previous state-of-the-art results by more than 26% in predicting the class of interest (death), achieving 87.1% of AUROC and macro F1 of 73.9%. We also show that some machine learning models can be very interpretable and reliable, yielding more accurate predictions while providing a good explanation for the ‘why’.

Conclusion

The best results were obtained using the meta-learning ensemble model – Stacking. State-of the art explainability techniques such as SHAP-values can be used to draw useful insights into the patterns learned by machine-learning algorithms. Machine-learning models can be more explainable than traditional statistics models while also yielding highly reliable predictions.