Prognostic models in first-episode psychosis – Authors' reply

We thank Daniel Whiting and Seena Fazel for their insightful comments about our work and important suggestions for improvement in our analyses. We completely agree that machine learning might not be appropriate for all prediction problems, though the distinction between what constitutes machine learning and what constitutes traditional statistical approaches is less clearcut, and some have suggested to consider them as being on a continuum.1Beam AL Kohane IS Big data and machine learning in health care.JAMA. 2018; 319: 1317-1318Crossref PubMed Scopus (407) Google Scholar We agree with Whiting and Fazel about models being clinical acceptable if prognostic factors are included as a starting point. However, given that a 2017 meta-regression did not find any significant predictors of remission in first-episode psychosis,2Lally J Ajnakina O Stubbs B et al.Remission and recovery from first-episode psychosis in adults: systematic review and meta-analysis of long-term outcome studies.Br J Psychiatry. 2017; 211: 350-358Crossref PubMed Scopus (114) Google Scholar one could argue that data-driven approaches, including machine learning, might offer a much better opportunity than traditional statistical approaches. Whiting and Fazel rightly point out the requirement of ten events per predictor parameter (EPP) for machine learning. We had 163 predictor parameters available for model development, which included most of the predictors that have been mentioned in the literature. We did not feel able to further reduce these variables on the basis of previous expert knowledge, because no consensus on reliable predictors of poor outcome in psychosis exists. We felt that it would be clinically less useful to combine individual items from rating scales into a single total score because this would necessitate carrying out the whole rating scale, causing the models to be non-parsimonious. We took a non-hypothesis driven approach to feature selection by using leave-one-site-out-cross-validation (LOSOCV) with elastic net for feature selection on our development data.3Zou H Hastie T Regularization and variable selection via the elastic net.J R Stat Soc Series B Stat Methodol. 2005; 67: 301-320Crossref Scopus (7921) Google Scholar For example, for the remission outcome for external validation in the OPUS study, our final logistic regression model had 22 predictors and was built on 673 participants with 320 events, giving a EPP of 14·5. We apologise if this method was not clear in our manuscript. Regarding the second point of Whiting and Fazel, we agree that measures of calibration and assessment of clinical usefulness are essential to evaluate a model's performance, along with measures of discrimination, before considering the application of any prediction model to clinical practice.4Steyerberg EW Vergouwe Y Towards better clinical prediction models: seven steps for development and an ABCD for validation.Eur Heart J. 2014; 35: 1925-1931Crossref PubMed Scopus (522) Google Scholar This is missing from our study and we thank Whiting and Fazel for highlighting this important limitation. It is one we will address in our future work. We declare no competing interests. Development and validation of multivariable prediction models of remission, recovery, and quality of life outcomes in people with first episode psychosis: a machine learning approachIn our machine learning analysis, we showed that prediction models can reliably and prospectively identify poor remission and recovery outcomes at 1 year for patients with first-episode psychosis using baseline clinical variables at first clinical contact. Full-Text PDF Open AccessPrognostic models in first-episode psychosisSamuel P Leighton and colleagues1 place a welcome focus on first-episode psychosis as an area in which prediction models could assist in developing treatment pathways on the basis of more individualised prognostic assessments. However, whether machine learning is the most appropriate approach, as well as the reporting and interpretation of model performance, will need clarification before these models can be clinically used. Full-Text PDF Open Access