Protocol: In multi-reader multi-case study designs, do AI-diagnostic tools improve accuracy outcomes for radiologists in cancer risk stratification?
Creators: name: Zhao, Dylan
name: Pluddemann, Annette
name: Jie, Xiaobo
name: Shahid, Muhammad Ayan
name: Qureshi, Bilal
name: Oke, Jason
Publisher: name: OSF Registries
Year: 2023
Subjects: subject: Medicine and Health Sciences
subject: Artificial Intelligence
subject: Cancer
subject: Computer Assistance
subject: Machine Learning
subject: Multi-reader Multi-Case
subject: Radiology

description: The use of artificial intelligence (AI) and machine learning (ML) techniques in healthcare has become a topic of significant discussion. Methods using such algorithms have demonstrated remarkable improvements in terms of accuracy and sensitivity in medical image recognition tasks1. In clinical practice, evidence has shown that AI technology can be used as a clinical decision tool to diagnose numerous diseases when using radiological techniques such as X-ray, CT, MRI, or ultrasound1. ML algorithms thus are seen as being able to efficiently provide diagnostically accurate data to patients on a shorter timescale. Challenges and complications Although AI has been lauded as a revolutionary technique and can be suggested to perform better than humans in healthcare on several issues regarding their accuracy and utility in clinical practice, some of the main issues that have arisen are: • Algorithms are normally inputted with a specific output in mind while a doctor may be able to make a more flexible decision. For example, an AI tool that has been developed to consider lymph node metastasis in pancreatic ductal adenocarcinoma may be able to outperform radiologists2. However, this model will not be able to then make decisions that only a doctor can on the best course of action for the patient. • Overdiagnosis3: The algorithms from AI detect and diagnose conditions whether they cause harm for a patient or not. For example, if we consider a CT pulmonary angiography (CTPA) to detect pulmonary emboli (PE) in the lungs, this is usually an incidental finding and even when detected, PE only needs to be treated on a symptomatic basis. AI tools for CTPA may detect this at a much higher rate. Are smaller lesions therefore being detected with the extra sensitivity that are less harmful? The question then arises on how we should approach this data: who should be treated and who should not? AI application in medical imaging analysis The future of AI seems to be as a diagnostic support tool integrated within a radiologist’s workflow, allowing an increase in efficiency and a reduction in errors1. Studies that thus compare AI models with radiologist or clinician-mediated diagnosis may not highlight the effects AI may have in a clinical setting and quantify the gain in accuracy from AI in reading images. More radiological test studies have thus shifted to implementing a multi-reader multi-case (MRMC) design4. This design is popular as once a radiologist has viewed twenty cases there is less information to be gained by asking them to view a further twenty than asking a different radiologist to view the same twenty5. The study design asks multiple radiologists to view the same cases and means that results can correlate within readers meaning that bootstrapping and modelling can account for these parameters to enhance statistical power. These studies’ primary outcome arises as a receiver operator characteristic (ROC) curve that describes a plot of sensitivity vs. 1-specificity for diagnostic tests6. The AUC can then be used as a measure to combine both sensitivity and specificity to compare diagnostic tools. In recent years, several systematic reviews regarding artificial intelligence and radiology have been published7,8. These have focussed on diagnostic performance and how AI models compare to a radiologist. Due to the heterogeneity of data across the field of AI tools these have often been narrative syntheses. By solely considering the MRMC study design this allows examination of a more homogenous dataset. The study design outlined above also will help interpretation of how AI-diagnostic tools may help optimise cancer management in healthcare. Studies using this as a primary outcome are often required by regulatory bodies for licensing new radiological devices9. Furthermore, by solely analysing MRMC studies this also will allow consideration of detailed methodological components across studies to consider data presentation and analysis within the field of AI and radiology. This systematic review thus aims to assess MRMC studies using ROC AUC outcome measures to consider whether AI-assisted clinician diagnosis may be beneficial. The review will also examine design, analysis, and reporting of MRMC studies regarding AI to consider how these may influence the data that is ultimately presented.