P0307 Application of a machine learning model in the 2 + 1 central reading paradigm to assess endoscopic severity in ulcerative colitis trials
Authors: Battioui C , Brodskiy P , Gottlieb K , Haft-Javaherian M , Eastman W , Lehrer J , Yu E , Onken D , Thomason D , Colucci D , Reinisch W , Baxi S 
Published: January 2025

Background

The 2 + 1 human central reader paradigm is the most common workflow to measure the endoscopy subscore, a component of the modified Mayo Score, in ulcerative colitis (UC) trials1. This workflow involves independent endoscopy video reviews by a local site investigator (LR) and central reader (CR1), with discrepancies typically addressed through an additional independent central read (CR2). Inter-rater disagreement in assessments remains a challenge, impacting study results2. Machine learning (ML) models have demonstrated the potential to aid in assessment standardization of the endoscopy subscore. Paradigms for the use of these models in endpoint assessments are currently under investigation. The objective of this study is to explore the application of an ML model to assess the LR and CR1 in the 2 + 1 central reader paradigm.

Methods

Endoscopy video recordings from the mirikizumab trials and routine practice were used to develop a state-of-the-art multistage deep learning algorithm to assess the endoscopy subscore on full-length UC videos. 639 videos (~25%) from the Phase 3 LUCENT1 (NCT03518086) induction trial for mirikizumab in UC (week 0 and 12 procedures) with a 2 + 1 centrally read endoscopy subscore per protocol, were held out to evaluate performance of the final locked model and this analysis. Videos in this cohort were randomly selected with a distribution of endoscopic severity similar to the overall study population (82.2% moderate-to-severe). Quadratic weighted kappa (QWK) was used to evaluate inter-rater agreement in pairwise comparisons between the ML model, LR, and CR1 assessments.

Results

QWK between ML model-assessed endoscopy subscore and the final 2 + 1 reference standard endoscopy subscore was 0.77 (95% confidence interval (CI) 0.74-0.80). In the baseline comparison between LR and CR1 assessments, the initial two reads in the 2 + 1 central reading paradigm, QWK was 0.71 (95% CI 0.67-0.75) and the agreement rate was 62.4%. Independent comparison of the ML model to the LR (ML vs LR) and CR1 (ML vs CR1) demonstrated similar findings: QWK was 0.69 (95% CI 0.64-0.73) and 0.72 (95% CI 0.68-0.76) and agreement rate was 59.5% and 66.5%, respectively.

Conclusion

We demonstrate that this ML model can be used effectively in the 2 + 1 central reading paradigm for evaluation of the endoscopy subscore in UC and automate one of the two initial assessments. The high discord between human reviewers creates a ceiling effect when comparing the ML model to any single human read, with performance measurements capped by the variability in the human read. Future studies will evaluate the ML model and human reviewers against alternative ground truth definitions to inform higher-quality paradigms for the assessment of endoscopic endpoints in trials.

References

Food and Drug Administration. Ulcerative Colitis: Developing Drugs for Treatment [Internet]. 2022 [cited 2024 Oct 17]. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/ulcerative-colitis-developing-drugs-treatment.

Wils, P., Jairath, V., Sands, B. E., Magro, F., Reinisch, W., Rubin, D., . . . & Peyrin-Biroulet, L. (2023). Comparison of treatment effect between phase 2 and phase 3 trials in patients with inflammatory bowel disease. United European Gastroenterology Journal, 11(8), 797-806.