Accurate, Robust, and Scalable Machine Abstraction of Mayo Endoscopic Subscores from Colonoscopy Reports
Authors: Silverman Anna L. , Bhasuran Balu , Mosenia Arman , Yasini Fatema , Gupta Saransh , Mardirossian Taline , Narain Rohan , Sewell Justin , Butte Atul J. , Rudrapatna Vivek A. 
Published: June 2022

Importance

Electronic health records (EHR) data are growing in importance as a source of evidence on real-world treatment effects. However, many clinical important measures are not directly captured as structured data by these systems, limiting their utility for research and quality improvement. Although this information can usually be manually abstracted from clinical notes, this process is expensive and subject to variability. Natural language processing (NLP) is a scalable alternative but has historically been subject to multiple limitations including insufficient accuracy, data hunger, technical complexity, poor generalizability, algorithmic unfairness, and an outsized carbon footprint.

Objective

Compare different algorithmic approaches for classifying colonoscopy reports according to their ulcerative colitis Mayo endoscopic subscores

Design

Other observational study – NLP algorithm development and validation

Setting

Academic medical center (UCSF) and safety-net hospital (ZSFG) in California

Participants

Patients with ulcerative colitis

Exposures

Colonoscopy

Main Outcomes and Measures

The primary outcome was accuracy in identifying reports suitable for Mayo subscoring (binary yes/no) and then separately assigning a Mayo subscore where relevant (ordinal). Secondary outcomes included learning efficiency from training data, generalizability, computational costs, fairness, and sustainability.

Results

Using automated machine learning (autoML) we trained a pair of classifiers that were 98% [91-99%] accurate at determining which reports to score and 97% [88-99%] accurate at assigning the correct Mayo endoscopic subscore. The binary classifiers trained on UCSF data achieved 96% accuracy on hold-out test data from ZSFG. Training these classifiers required 4 hours of computation on a standard laptop. Classification errors were not associated with either gender or area deprivation index. The carbon footprint of this approach was 24x less than current deep learning algorithms for clinical text classification.

Conclusions and Relevance

We identified autoML as an efficient and robust method for training clinical text classifiers. AutoML-trained classifiers demonstrated many favorable properties including generalizability, limited effort needed for data annotation and algorithm training, fairness, and sustainability. More generally, these results support the feasibility of using unstructured EHR data to generate real-world evidence and drive continuous improvements in learning health systems.

Key Points

Question

Is natural language processing (NLP) a viable alternative to manually abstracting disease activity from procedure notes?

Findings

We compared different methods for abstracting the ulcerative colitis Mayo endoscopic subscore from colonoscopy reports. Classifiers trained using automated machine learning (autoML) achieved the greatest accuracy (97%), recognized when to abstain, generalized well to other health systems, required limited effort for annotation and programming, demonstrated fairness, and had a small carbon footprint.

Meaning

NLP methods like autoML appear to be sufficiently mature technologies for clinical text classification, and thus are poised to enable many downstream endeavors using electronic health records data.