P051 Development and evaluation of a text-analytics algorithm for automated application of the BSR COVID-19 shielding criteria in rheumatology patients
Authors: Jani Meghna , Alfattni Ghada , Belousov Maksim , Zhang Yuanyuan , Cheng Michael , Webb Karim , Kanter Andrew , Dixon William , Nenadic Goran 
Published: April 2023

Abstract

Background/Aims
In April 2020 the British Society for Rheumatology (BSR) issued a risk stratification guide to identify patients at the highest risk of COVID-19 requiring shielding. This guidance was based on patients’ age, comorbidities, and immunosuppressive therapies - including biologics that are not captured in primary care records. This meant rheumatologists needed to manually review outpatient letters to score patients’ risk. The process required considerable clinician time, with shielding decisions not always transparently communicated. Our aim was to develop an automated shielding algorithm by text-mining outpatient letter diagnoses and medications, reducing the need for future manual review.

Methods
Rheumatology outpatient letters from Salford Royal Hospital, a large UK tertiary hospital, were retrieved between 2013-2020. The two most recent letters for each patient were extracted, created before 01.04.2020 when BSR guidance was published. Free-text diagnoses were processed using Intelligent Medical Objects® software1 (Concept Tagger), which utilised interface terminology for each condition mapped to a SNOMED-CT code. We developed the Medication Concept Recognition tool (MedCore Named Entity Recognition) to retrieve medications type, dose, duration and status (active/past) at the time of the letter. The medication status was established based on the heading where they appeared (e.g. past medications, current medications), but incorporated additional information such as medication stop dates. The age, diagnosis and medication variables were then combined to output the BSR shielding score. The algorithm’s performance was calculated using clinical review as the gold standard.

Results
To allow for the comparison with manual decisions, we focused on all 895 patients who were reviewed clinically. 64 patients (7.1%) had not consented for their data to be used for research as part of the national opt-out scheme. After removing duplicates, 803 patients were used to run the algorithm. 11,558 free-text diagnoses were extracted and mapped to SNOMED CT, with 15,003 free-text medications (that included past, present and any planned treatment). The automated shielding algorithm demonstrated a sensitivity of 80.3% (95% CI: 74.7, 85.2%) and specificity of 92.2% (95% CI: 89.7, 94.2%). Positive likelihood ratio was 10.3 (95% CI: 7.7, 13.7), negative likelihood ratio was 0.21 (95% CI: 0.16, 0.28), F1 score was 0.81. False positive rate was 7.9%, whilst false negative rate was 19.7%. Further evaluation of false positives/negatives revealed clinician interpretation of BSR guidance and misclassification of medications status were important contributing factors.

Conclusion
An automated algorithm for risk stratification has several advantages including reducing clinician time for manual review to allow more time for direct care, improving efficiency and transparently communicating decisions based on individual risk. With further development, it has the potential to be adapted for future public health initiatives that requires prompt automated review of hospital outpatient letters.
Disclosure
M. Jani: None. G. Alfattni: None. M. Belousov: None. Y. Zhang: None. M. Cheng: None. K. Webb: None. A. Kanter: None. W. Dixon: Consultancies; Google (unrelated to this work). G. Nenadic: None.