External validation of inpatient neonatal mortality prediction models in high-mortality settings
Authors: Tuti Timothy , Collins Gary , English Mike ,   on behalf of the Clinical Information Network, Mbevi George , Wainaina John , Mumelo Livingstone , Gicheha Edith , Muinga Naomi , Ogola Muthoni , Oyiengo Laura , Mwangi Caroline , Were Fred , Vitalis Juma , Bonface Nyumbile , Malangachi Roselyne , Manyasi Christine , Mutinda Catherine , Kimutai David Kibiwott , Aden Rukia , Emadau Caren , Jowi Elizabeth Atieno , Muithya Cecilia , Nzioki Charles , Tunje Supa , Musyoka Penina , Mwangi Wagura , Mithamo Agnes , Kuria Magdalene , Njiru Esther , Ngina Mwangi , Mwangi Penina , Inginia Rachel , Musabi Melab , Namulala Emma , Ochieng Grace , Thuranira Lydia , Makokha Felicitas , Ojigo Josephine , Maina Beth , Mutinda Catherine , Waiyego Mary , Lusweti Bernadette , Ithondeka Angeline , Barasa Julie , Liru Meshack , Kibaru Elizabeth , Nyaribari Alice Nkirote , Akuka Joyce , Wangari Joyce , Ngoda Amilia , Emenwa Aggrey Nzavaye , Mochache Dolphine , Wesakania Patricia Nafula , Lipesa George , Mbungu Jane , Mutenyo Marystella , Mbogho Joyce , Baswetty Joan , Jambi Ann , Aritho Josephine , Njambi Beatrice , Mucheke Felisters , Kioni Zainab , Kinyua Lucy , Kethi Margaret , Oguda Alice , Situma Salome Nashimiyu , Gachaja Nancy , Mwangi Loise N. , Mwai Ruth , Muruga Irginia Wangari , Mburu Nancy , Muteshi Celestine , Bwire Abigael , Muyale Salome Okisa , Situma Naomi , Mueni Faith , Mwaura Hellen , Mututa Rosemary , Lavu Caroline , Oketch Joyce , Olum Jane Hore , Nyakina Orina , Njeru Faith , Chelimo Rebecca , Mwaura Margaret Wanjiku , Wambugu Ann , Mburu Epharus Njeri , Tindi Linda Awino , Akumu Jane , Otieno Ruth , Osok Slessor , Kulubi Seline , Wanjala Susan , Njeru Pauline , Mbogo Rebbecca Mukami , Ollongo John , Soita Samuel , Mirenja Judith , Nguri Mary , Waweru Margaret , Oruko Mary Akoth , Kuya Jeska , Muthuri Caroline , Muthiani Esther , Mwangi Esther , Nganga Joseph , Tanui Benjamin , Wanjau Alfred , Onsongo Judith , Muigai Peter , Namayi Arnest , Kosiom Elizabeth , Cherop Dorcas , Marete Faith , Simiyu Johanness , Danga Collince , Oyugi Arthur Otieno , Okoth Fredrick Keya , Aluvaala Jalemba 
Published: August 2022

Abstract
Background
Two neonatal mortality prediction models, the Neonatal Essential Treatment Score (NETS) which uses treatments prescribed at admission and the Score for Essential Neonatal Symptoms and Signs (SENSS) which uses basic clinical signs, were derived in high-mortality, low-resource settings to utilise data more likely to be available in these settings. In this study, we evaluate the predictive accuracy of two neonatal prediction models for all-cause in-hospital mortality.

Methods
We used retrospectively collected routine clinical data recorded by duty clinicians at admission from 16 Kenyan hospitals used to externally validate and update the SENSS and NETS models that were initially developed from the data from the largest Kenyan maternity hospital to predict in-hospital mortality. Model performance was evaluated by assessing discrimination and calibration. Discrimination, the ability of the model to differentiate between those with and without the outcome, was measured using the c-statistic. Calibration, the agreement between predictions from the model and what was observed, was measured using the calibration intercept and slope (with values of 0 and 1 denoting perfect calibration).

Results
At initial external validation, the estimated mortality risks from the original SENSS and NETS models were markedly overestimated with calibration intercepts of − 0.703 (95% CI − 0.738 to − 0.669) and − 1.109 (95% CI − 1.148 to − 1.069) and too extreme with calibration slopes of 0.565 (95% CI 0.552 to 0.577) and 0.466 (95% CI 0.451 to 0.480), respectively. After model updating, the calibration of the model improved. The updated SENSS and NETS models had calibration intercepts of 0.311 (95% CI 0.282 to 0.350) and 0.032 (95% CI − 0.002 to 0.066) and calibration slopes of 1.029 (95% CI 1.006 to 1.051) and 0.799 (95% CI 0.774 to 0.823), respectively, while showing good discrimination with c-statistics of 0.834 (95% CI 0.829 to 0.839) and 0.775 (95% CI 0.768 to 0.782), respectively. The overall calibration performance of the updated SENSS and NETS models was better than any existing neonatal in-hospital mortality prediction models externally validated for settings comparable to Kenya.

Conclusion
Few prediction models undergo rigorous external validation. We show how external validation using data from multiple locations enables model updating and improving their performance and potential value. The improved models indicate it is possible to predict in-hospital mortality using either treatments or signs and symptoms derived from routine neonatal data from low-resource hospital settings also making possible their use for case-mix adjustment when contrasting similar hospital settings.