Leadership Focus on Advancing Cancer Research and Treatment: A Conversation Between Johannes Czernin, Caius Radu, and Antoni Ribas.

1178 Objectives: In recent years the medical imaging field has seen a rapid increase in machine learning (ML) use for computer-assisted interpretation. One of the limitations of ML is its reliance on large and high-quality labeled data sets. This limitation can be alleviated if the metadata of patients relevant to the disease studied can be accurately extracted. However, most patient metadata stored in text reports are not directly suitable for ML. To address this, time-consuming manual data abstraction is often employed. Information Extraction (IE) is a Natural Language Processing (NLP) technique to automatically extract information from text. IE it can facilitate or possibly replace manual abstraction. In this study, we examine the utility of IE in a weak-labeling pipeline to extract perfusion interpretation metadata for a given diagnosis from unstructured clinical text reports of positron emission tomography (PET) myocardial perfusion imaging (MPI). Methods: The study population consisted of 15,550 patients referred for Rb-82 PET MPI. The information extracted from the perfusion reports included clinical standard diagnosis for detection (normal/abnormal) and localization (LAD, LCX, RCA vessel) of coronary artery disease (CAD) as scar and/or ischemia. To extract relevant labels, a NLP-spaCy engine was developed using a clinically relevant language model (sciSpaCy, Allen AI) and customized negation detection (negspaCy). The NLP engine was wrapped in a ‘weak label’ pipeline (Snorkel) to increase extraction accuracy. Weak-labeling pipelines apply an ensemble approach to the plateau in accuracy often seen when using a single machine-generated or ‘weak label’ like IE. The premise of the pipeline is that the combination of multiple weak label rules can result in higher classification accuracy than the use of single-label extraction rules. Labeling rules were developed to examine the token and entity levels to identify syntactic (language model) linkages of dual labels, as well as non-negated single labels. Individual labeling rules were evaluated using 3 metrics: i. coverage or the percentage of data points covered by the label, ii. overlaps or the fraction of data points with at least two agreeing labels, and iii. conflicts or the fraction of data points with conflicting labels. The overall label generated by a majority voting model was then evaluated for precision and recall against a manually validated data set. Results: Single-label data extraction (Abnormal, Scar, Ischemia) produced F1 scores between 98% and 99%. Extraction of two associated (dual) labels achieved F1 scores between 90% and 98%, for scar and ischemia in the 3 vessel territories. Recall was generally higher for these dual-label extractions, indicating the model’s ability to strongly distinguish true positives from false positives but may not identify all true positives. Conclusions: Ensemble ‘weak label’ pipelines produced high levels of classification accuracy needed for clinical relevance. In single-label extractions, manual data extraction and syntactic based rules mirrored each other in performance, providing verification of the machine-generated metadata’s internal consistency. For dual-label extractions, there was greater variability in coverage, overlap and conflicts for a given rule set. This is in line with the lower overall performance levels, but the combinatorial power of the ensemble results still maintained high levels of classification accuracy. In particular, there was lower overall performance of multiple label assessment of scar/ischemia in the LCX, which is consistent with an observed variability in reporting for this vessel territory. These methods have potential for use in decision support tools for computer-assisted interpretation by automating the process of clinical text abstraction for relevant metadata. Future studies will examine the utility of these methods in extracting additional clinical text features needed for ML modeling.