Title: Can automated Diabetic Retinopathy Image Assessment softwares replace one or more steps of manual imaging grading and is this cost-effective for the NHS Diabetic Eye Screening Programme?
Funder: Department of Health (England)

Abstract: DESIGN: An observational study to quantify screening performance diagnostic accuracy and cost-effectiveness of automated primary grading in routine English practice. Three automated software programmes will be assessed. Study phase 1 is retrospective: the automated software packages will grade 18000 screening episodes which have been manually graded from patients attending diabetic retinopathy (DR) screening programme. Phase 2 is a prospective implementation study of an additional 6000 screening episodes from the same clinic. Manual grading will be 2-field-2 step by human graders who meet English National Screening Programme (ENSPDR) quality assurance standards and is the reference standard for this study.SETTING: Diabetic eye screening service in England compliant with ENSPDRstandards.TARGET POPULATION: Diabetic patients over 12 years of age eligible for DR screening attending a London PCT screening programme with a wide spectrum of ages and ethnicity(24%South Asian 17% black African/Carribean and 38%white European) The retrospective study (n=18000 consecutive screening episodes) will begin 12 months prior to the start of the grant. The prospective study (n=6000 screening episodes) will begin soon after the grant starts.HEALTH TECHNOLOGIES BEING ASSESSED: Commercially available software with CE mark or likely to obtain CE mark within 6 months of study: iGrading (Medalytix) Retmarker (Critical-Health) EyeCheck (IDx). All investigators are independent of commercial software development and have no commercial interest or IP in automated DR screening. Uniquely we already have access and support for all 3 software packages for the study and exclusive access to the IDx software for evaluation in the UK.MEASUREMENT OF COST AND OUTCOMES: Each software provides a binary output of “no disease” or “disease present”. Cost effectiveness will be evaluated based on the expected cost per true positive case of retinopathy and cost per QALY. Measures of performance of each automated grading system will include sensitivity false positive rate (1-specificity) and likelihood ratios using the final manual grade as the reference standard. These measures will be determined for referrable DR overall and for each grade of DR. Diagnostic accuracy will be described by 95% confidence intervals (CI) for each primary outcome. Clinical interpretation of these measures will focus on the lower limit of the 95%CI for the detection of proliferative (R3) pre-proliferative (R2) DR grades and maculopathy (M1). This indicates the potential number of screening episodes missed by the automated systems requiring clinical intervention. The upper CI for false positive rates for DR grades R1(background) and R0 (no DR) will assess the additional number of screening episodes requiring manual grading and the risk of over-diagnosis. Technical failure rate will be determined. Secondary outcomes will measure agreement for automated systems and manual grading against the final clinical decision for a screening episode. Disagreements between the human grader and one or more automated systems will be arbitrated on a subset of images from the 18000 retrospective screening episodes.by an internationally recognised fundus photographic reading centre masked to the original gradings. This gives an independent assessment of the relative performance of manual and automated systems and provides external validity to the project. Emphasis will be placed on arbitrating discrepancies with manual grades R3 R2 M1 and a proportion of R1 and R0 grades. Technical failure rates for automated and manual grading will be compared. (Continued in ‘additional information section)


Research activity codes: 6.3 MEDICAL DEVICES;7.3 MANAGEMENT AND DECISION MAKING
Health categories: EYE;METABOLIC AND ENDOCRINE