S0818 Accurate Machine Classification of Ulcerative Colitis Mayo Subscores From Electronic Health Record Procedure Reports (2020)

INTRODUCTION: Electronic health records data are receiving attention as a potential source of evidence on real-world treatment effects. However, these data typically do not explicitly capture validated disease activity scores, limiting their utility. Although such scores are often abstractable from clinical free-text, this manual process is laborious, subject to variability, and requiring of domain expertise. Here, we report the performance of several machine learning models trained to classify Ulcerative Colitis (UC) procedure reports according to the Mayo endoscopic subscore. This ordinal subscore ranges from 0 to 3 in order of increasing UC activity. METHODS: In March 2020 we queried the electronic health records at our institution for all lower endoscopy reports done in patients with a history of an IBD diagnosis code. A gastroenterologist classified a random sample of 700 reports according to their suitability for Mayo subscoring: those with a clear diagnosis of UC and surgically unaltered anatomy. These reports were then assigned Mayo subscores based on the most severely affected segment with any friability scored as a 2 or 3. For both text classification tasks, we trained and evaluated several machine learning models using 5-fold cross-validation. Within each fold we performed text pre-processing, word vectorization, and model training followed by evaluation using the validation data. We assessed model performance using several common metrics for binary and multinomial classification. Models were compared to a null model (negative control) and a second blinded annotator (positive control). RESULTS: On the task of identifying reports suitable for Mayo subscore classification, multiple models attained nearly perfect performance, with the best achieved by random forest and ensemble models (97-98% accuracy). The accuracy of the second human annotator was 95.0% (Table 1) Among reports suitable for Mayo subscore classification, the best-performing model was a random forest classifier (0.156 mean absolute error, 87.2% accuracy). The comparable performance of the second human annotator for this task was 0.137 and 83.7% respectively (Table 2). CONCLUSION: Machine classifiers can identify and score UC procedure reports at a level of accuracy equal to humans. Future studies are needed to assess model generalizability and to use these results in support of real-world effectiveness investigations in IBD. The analytic code for model training is available as open-source software.Table 1.: Summary of model performance on the task of classifying Ulcerative Colitis from non-Ulcerative Colitis procedure reports. All machine learning model metrics correspond to the mean across cross-validated folds. The null model predicts the most common report type for all reportsTable 2.: Summary of model performance on the task of classifying mayo endoscopic subscores from ulcerative colitis procedure reports. All machine learning model metrics correspond to the mean across cross-validated folds. Null Model 1 predicts the most common Mayo subscore (0) for all reports (maximizing accuracy). Null Model 2 predicts a Mayo subscore of 1 for all reports (minimizing mean absolute error)