Prognostic predictions in psychosis: exploring the complementary role of machine learning models
Authors: van Dee Violet , Kia Seyed Mostafa , Fregosi Caterina , Swildens Wilma E. , Alkema Anne , Batalla Albert , van den Berg Coen , Coric Danko , van Dellen Edwin , Dijkstra Lotte G. , van den Doel Arthur , Dominicus Livia S. , Enterman John , Gerritse Frank L. , van der Horst Marte Z. , van Houwelingen Fedor , Koch Charlotte S. , Koomen Lisanne E.M. , Kromkamp Marjan , Lancee Michelle , Mouthaan Brian E. , van Rappard Diane F. , Regeer Eline J. , Salet Raymond W.J. , Somers Metten , Straalman Jorgen , de Vette Marjolein H.T. , Voogt Judith , Winter-van Rossum Inge , Kahn René S. , Cahn Wiepke , Schnack Hugo G. 
Published: February 2025

BACKGROUND

Predicting outcomes in schizophrenia spectrum disorders is challenging due to the variability of individual trajectories. While machine learning (ML) shows promise in outcome prediction, is has not yet been integrated into clinical practice. Understanding how ML models (MLMs) can complement psychiatrists’ predictions and bridge the gap between MLM capabilities and practical use is key.

OBJECTIVE

This study aims to compare the performance of psychiatrists and MLMs in predicting short-term symptomatic and functional remission in patients with first-episode psychosis and explore whether MLMs can improve psychiatrists’ prognostic accuracy.

METHOD

Twenty-four psychiatrists predicted symptomatic and functional remission probabilities based on written baseline information from 66 patients in the OPTiMiSE trial. ML-generated predictions were then shared with psychiatrists, allowing them to adjust their estimates. A questionnaire assessed trust in MLMs, perceived information gaps, and psychiatrists’ self-assessed predictive accuracy, which was compared to actual accuracy.

FINDINGS

The predictive accuracy of the MLM was comparable to that of psychiatrists for symptomatic remission (MLM: 0.50, psychiatrists: 0.52) and functional remission (MLM: 0.72, psychiatrists: 0.79). Interrater agreement was low but comparable for psychiatrists and the MLM. Although the MLM did not improve overall predictive accuracy, it showed potential in aiding psychiatrists with difficult-to-predict cases. However, psychiatrists struggled to recognize when to rely on the model’s output and we were unable to determine a clear pattern in these cases based on their characteristics. Psychiatrists could not reliably estimate their predictive accuracy. Psychiatrists expressed moderate to high trust in MLMs for prognostic prediction, but highlighted concerns about the lack of transparency and interpretability of model outputs.

CONCLUSIONS

MLMs are a promising tool for supporting psychiatric decision-making, particularly in challenging cases. However, their potential remains underutilized due to limitations in predictive accuracy and a lack of clarity in how predictions are generated. Addressing these issues is essential to build trust and foster integration into clinical practice.

CLINICAL IMPLICATIONS

MLMs are best suited as supplementary tools, providing a second opinion while psychiatrists retain decision-making autonomy. Integrating predictions from both sources may help reduce individual biases and improve accuracy. This approach leverages the strengths of MLMs without compromising clinical responsibility.

SUMMARY BOX

What is already known on this topic

While machine learning models (MLMs) show promise in predicting outcomes in psychotic disorders, they have yet to be integrated into clinical practice. Evidence on the predictive accuracy of psychiatrists for these disorders is limited, with only two small studies published before 1990 suggesting moderate accuracy. Comparisons of MLMs and psychiatrists in this context have not been previously conducted.

What this study adds

This is the first study to compare the predictive accuracy of psychiatrists with that of an MLM for psychotic disorders and to assess whether an MLM can enhance psychiatrists’ performance. It highlights that while MLMs do not improve overall accuracy, they may support psychiatrists in difficult cases. Insights into psychiatrists’ trust in MLMs and the challenges of implementing these models are also provided.

How this study might affect research, practice, or policy

The findings emphasize the need for advancements in MLM accuracy, interpretability, and strategies to identify cases where MLMs are most beneficial. These improvements could foster effective integration of MLMs as supplementary tools in clinical practice, aiding psychiatrists in decision-making while maintaining their autonomy.