Early triage of critically ill COVID-19 patients using deep learning
Authors: Jianxing He
Niyun Zhou
Jianhua Yao
Shiyue Li
Wei Wang
Ling Sang
Nuofu Zhang
Wenhua Liang
Qingquan Lv
Nanshan Zhong
Hengrui Liang
Mark Zanin
Fan Yang
Ailan Chen
Limin Ou
SookSan Wong
Xiao Han
Jun Guo
Jiatao Lu
Yi Zhao
Hanbo Chen
Wenjing Huan
Ligong Lu
Rong Zhou
Zisheng Chen
Guoqiang Chen
Weimin Tang
Jun Liu
Yuanda Xu
Junzhou Huang
Yimin Li
Haiyan Guo
Weijie Guan
Journal: Nature Communications (2020)
Discipline: Multidisciplinary
Keywords: COVID-19; deep learning; survival model; critical illness; patient triage
Affiliations: China State Key Laboratory of Respiratory Disease and National Clinical Research Center for Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China., Tencent AI Lab, Shenzhen, China., Hankou Hospital, Wuhan, China., School of Public Health, The University of Hong Kong, Hong Kong SAR, China., Department of Thoracic Surgery, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China., Department of Intensive Care Unit, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China., Foshan Hospital, Foshan, China., Daye Hospital, Hubei, China., Tencent Healthcare, Shenzhen, China., Department of Respiratory Disease, The Sixth Affiliated Hospital of Guangzhou Medical University, Guangzhou, China., Zhuhai People Hospital, Zhuhai, China., China State Key Laboratory of Respiratory Disease and National Clinical Research Center for Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China. ., Tencent AI Lab, Shenzhen, China. .

Abstract:  The sudden deterioration of patients with novel coronavirus disease 2019 (COVID-19) into critical illness is of major concern. It is imperative to identify these patients early. We show that a deep learning-based survival model can predict the risk of COVID-19 patients developing critical illness based on clinical characteristics at admission. We develop this model using a cohort of 1590 patients from 575 medical centers, with internal validation performance of concordance index 0.894 We further validate the model on three separate cohorts from Wuhan, Hubei and Guangdong provinces consisting of 1393 patients with concordance indexes of 0.890, 0.852 and 0.967 respectively. This model is used to create an online calculation tool designed for patient triage at admission to identify patients at risk of severe illness, ensuring that patients at greatest risk of severe illness receive appropriate care as early as possible and allow for effective allocation of health resources. 

Retracted: No

PubPeer comments:
 This prediction model is too  similar with another model developed by the same research group, which was already published in JAMA Internal Medicine. https: jamanetwork.com journals jamainternalmedicine fullarticle 2766086 Both models use the identical dataset, and predict the same outcome (critical illness). The only difference is the JAMA paper used logistic regression, and this Nature Communications paper used Cox model. Developing prediction models with different algorithms, and submitting to different journals as TWO models, without referring another model (especially within the same research group), looks like duplicate publication. What is your opinion, peers? , 
 #1 This is Wenhua Liang, the first author of this study. Thank the peers for the inquiries about our recent paper u201cEarly Triage of Critically-Ill COVID-19 Patients Using Deep Learningu201d (NCOMMS-20-15205-T). We understand and respect the principle of novelty and uniqueness of scientific work. The current NCOMMS paper differed with another previously published paper in JAMA internal med regarding purpose, data and methodology which we have more fully elucidated below: 1) The clinical purpose is different. In the JAMA-IM paper, we have only the binary outcome (yes no), so a logistic model to predict whether or not a patient will develop into critically illness during the hospital stay was established. However, the speed at which the patient will develop critical illness will also influence the decision making. Thus, we further requested the time information from the data center to establish a time dependent model based on deep learning Cox regression. We herein generated a time-dependent prediction to predict the risk at different time points (i.e. 5 days, 10 days, 30 days etc.) and thus, allowing better resource management, whereas the prediction in the JAMA-IM paper is yes no at one point. In addition, a visualized nomogram was provided to make our model explainable. 2) The scientific purpose is different. We aimed to use deep learning methods to build a disease prognostic model and proved that it outperforms a traditional linear model. In addition, based on the deep learning methods, we proposed a new imputation scheme to handle missing data automatically, making prediction more widely available in different areas and more robust for errors in data acquisition. This function is not available in the JAMA-IM model and this is one of the reasons why we performed this further study. These themselves are novel breakthrough. 3) The dataset is different. We used a major dataset managed by the China national health commission, which was released by some previous studies, to build the model in Nat Commun and JAMA-IM paper. However, in the new investigation in Nat Commun, we collected a new cohort in Wuhan and added data not only at hospital admission but also after hospital admission with monitoring data (3-5 timepoints for each new collected case), and used this data for further tracking and validation. Based on this, we proposed a risk monitoring model using monitoring data after hospital admission, which is also a novel idea. This part of the model and data was not available in the JAMA-IM paper. Furthermore, the external validation cohort was almost doubled in the new paper (increased from 710 to 1393). In the early outbreak of COVID-19, data collection was more difficult but research on this subject is more urgent than usual. It is almost impossible and not reasonable to perform research with brand new non-overlapping data. 4) The reason we did not cite the JAMA-IM paper in the current paper was the unmatched timing. The JAMA-IM paper was finally published in May, when the current work in Nat Commun was in its final review, thus we did not change the logic of description, considering the above differences. Thank you again for the interest in our studies and the efforts in maintaing academic principles. , 
 #2 Thanks for the clarification from the first author, Liang. However, I have different opinions on some of the points in his her reply. 1. The clinical purpose is NOT different, the clinical purpose in both papers was predicting critical illness, which is SAME . If the definition of endpoint was same (this is clear from the papers, and can be confirmed by having the same number of 131 events in both studies), no matter what model was used. Treating the endpoint as a binary outcome (with logistic model as in JAMA-IM) or time-to-event outcome (with Cox model as in Nat Commun) is just a choice in model development, which will not lead to different clinical purpose either. Compared to logistic model, Cox model did have some advantages (e.g. it can predict the risk at different time points, as the author mentioned above), but it does not mean one group can submit a sub-optimal model with logistic model and a improved Cox model almost at same time as two original studies. In the end, the additional nomogram is only for model presentation implementation, and with without a nomogram can not be seen as a difference. 2. The scientific purpose is NOT different. Indeed, comparing to the JAMA-IM paper, this Nat Commun paper does have some improvements. But as already mentioned in Point 1, using different algorithms is not evidence of being separate studies. If the authors argument holds, then one group can use logistic model, Cox model, machine learning models, deep learning techniques, etc. to develop as many  different models as they like, and submit many articles as different studies. Is it acceptable from scientific point of view? Although  deep learning was mentioned in the title and main text as the selling point, the final model was a just a normal Cox model. But this is not important at all given the definition of  different as discussed above. If the authors believe the model proposed in Nat Commun outperforms the JAMA-IM, they should only submit the preferred one, but not to submit both at the same time period. 3. The model derivation dataset is same. The exactly same dataset was used to build the model in Nat Commun and JAMA-IM paper, this can be seen from the number of centers (575), number of patients (1590) and number of events (131). The so called  risk monitoring model is just doing the prediction with monitoring data, with the baseline model, which will not lead to a new model either. 4.  The reason we did not cite the JAMA-IM paper in the current paper was the unmatched timing. The JAMA-IM paper was finally published in May, when the current work in Nat Commun was in its final review, thus we did not change the logic of description, considering the above differences. Does this mean, the two studies were performed and submitted almost at the same time? If the authors do think the Nat Commun improves what they proposed in JAMA-IM, they should cite the JAMA-IM paper and give cautions to model users not to use the JAMA-IM model. Or they can withdraw the JAMA-IM paper first, before getting the Nat Commun published. The fact is, in JAMA-IM paper, the authors did not mention they are currently working on an improved model; while in the Nat Commun paper, they did not mention their previous work and point out the rationale for this redevelopment. This can not be explained as  the unmatched timing . The final question is, if they are indeed two different models, in which situation you recommend to use JAMA-IM model, and in which situation you recommend Nat Commun model? If only one is enough, why bother publishing two papers? , 
 #3 Dear Dr. Orthotylus Serus, thank you for providing different perspectives. Although we have some discrepancies in the understanding the academic novelty, I believed your comments merit a serious discussion within our research team. Regarding the last question that which model the colleagues should choose or we recommend, which is very practical and interesting, our recommendation right now is to present both models to the colleagues and let them decide which prediction result to adopt, or refer to both to gain more comprehensive prediction and cross validate. One point to add is the variables being included finally were not exactly the same of the two models, due to the different methodology. And the Nat Commun model is not necessarily better than the one in JAMA-IM. I would like to use a simple example to express our viewpoint, just like Window 7 and Window 10 systems, Win 10 has more functions than Win 7, but some people still would choose Win 7 over Win 10 because of proficiency and better compatibility with his her computer. Thank you again for raising these interesting questions which made us think more insightfully. , 
 #4 Thanks for the very interesting discussion, as well as the example. Please note that, Windows 7 was on market in 2009 and Windows 10 was launched in 2015, and most important, everyone knows Windows 10 is an updated version of the Windows system. But for these two papers under discussion, the models were developed in the same time period and published as independent studies in two journals without citing referring to the other. That is the definition of duplicate publication. The authors still did not provide a reasonable explanation for this. And selling a software is different than doing scientific researches: one can decide to buy Win 10 or stay with Win 7, but if both your papers got published, actually you have already sold the same product twice. Editors, reviewers, readers and users need to pay for the bill (i.e. spending their time for a duplicate study). It is not good conduct in scientific research. The methodology of development and validation of clinical prediction models includes study design, data selection, variable selection, model fitting, etc. The algorithm employed in a prediction model is only a small part of the methodology. So I do not agree with the authors argument that they used  different methodology , if only the algorithms are different. The author did not provide a direct answer to Point 2. If you had also tried Random Forest model, are you ready for a THIRD paper? Last but not least, this is the first time I see one research group publish two papers based on same data and study the same outcome, and let the readers to choose one by themselves. It makes no sense at all. , 
 The biggest issue here is the authors may have taken advantages by omitting the information that the other paper was being under-review by another journal and misleading the editors to consider the submission as an unique original research. The editors are the best ones to make a conclusion for this discussion. It might be helpful if the editors from JAMA Internal Medicine and Nature Communications start an investigation and check the submission cover letters. It will give a clear answer whether the authors properly mentioned the other paper to the editors from both journals. , 
 #1 If you have a look at the https: www.nejm.org doi full 10.1056 nejmoa2002032, you may surprise that the data had been previously reported. Check the timeline. ,