Assessing the Impact of Using Autonomous Artificial Intelligence (AI) for Pre-screening of Diabetic Retinopathy (DR) and Diabetic Macular Edema on Physician Productivity in Bangladesh
Phase: NA
Conditions: Diabetic Retinopathy, Diabetic Macular Edema
Sponsor: Orbis

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Bangladesh PRODUCTIVity in Eyecare (B-PRODUCTIVE) Trial 
 
NCT05182580 
 
Protocol  
 
August 1, 2022 
 
 
 

Final Protocol  
 
Bangladesh PRODUCTIVity in Eyecare (B-PRODUCTIVE) Trial 
 
Sponsor: Orbis 
Collaborators: Digital Diagnostics Inc. and Deep Eye Care Foundation (DECF) 
Study Aim: To assess the impact of using autonomous artificial intelligence (AI) for 
identification of diabetic retinopathy (DR) and diabetic macular edema on productivity of 
retina specialists in Bangladesh. 
Hypothesis: Autonomous AI increases retina specialist productivity 
Main Study Question: Will retina specialists complete a greater number of diabetic eye 
exams per working hour (including persons reviewed by AI whom the retina specialist 
does not need to see personally) when they use autonomous AI in a randomized clinical 
trial? 
Design: Cluster-randomized (by clinic day) controlled trial. 
Randomization: By clinic day. Each morning the clinic manager will be notified if it is an 
Intervention (AI) or Control (non-AI) day. 
Interventions: All patients in both groups go through the eligibility checklist. If 
approved, they will be evaluated by autonomous AI. This is done to decrease potential 
bias (neither patients nor physicians know the group assignment of participants) and 
concealment (so that neither patients nor doctors can arrange visits on a known 
"Intervention Day"). 
Intervention Group: On randomly selected "Intervention" clinic days, if patients screen 
positive or have insufficient image quality, they continue to the retina specialist. If not 
eligible for autonomous AI, they proceed straight to the retina specialist without 
autonomous AI evaluation. If patients receive a negative result, they do not see the 
retina specialist, and are referred for a visit at the regular eye clinic (not the retina clinic) 
in 3 months. 
Control Group: On randomly-selected "Control Days," all patients see the retina 
specialist, irrespective of the results of autonomous AI evaluation. 
Masking: The retina doctors are masked both patient group assignment (that is, 
whether autonomous AI was used for pre-screening or not on the particular clinic day) 

and also masked to the results of the AI on Intervention days. Patients are also masked 
to group assignment and autonomous AI results. 
 
Study Design  
Study Type: Interventional (Clinical Trial) 
Estimated Enrollment: 924 participants 
Allocation: Randomized 
Intervention Model: Parallel Assignment 
Intervention Model Description: Cluster-randomized (by clinic day) controlled trial. 
Masking: Double (Participant, Patient) 
Masking Description: The retina specialists are masked both to patient group 
assignment (that is, whether autonomous AI results were used or not on the particular 
clinic day) and also masked to the results of the autonomous AI on Intervention days. 
Patients are also masked to group assignment and autonomous AI screening results. 
Primary Purpose: Diagnostic 
Official Title: Assessing the Impact of Using Autonomous Artificial Intelligence (AI) for 
Pre-screening of Diabetic Retinopathy (DR) and Diabetic Macular Edema on Physician 
Productivity in Bangladesh 
Actual Study Start Date: March 20, 2022 
Estimated Primary Completion Date: December 2022 
 
Arms and Interventions 
 
Arm 
Intervention 
Experimental: Intervention group 
Autonomous AI results are used to 
evaluate if the participant needs to see 
the retina specialist (positive result0 or 
not (negative result).  
Diagnostic test: results utilized from 
autonomous AI diagnostic system for 
diabetic retinopathy and/or diabetic 
macular edema.  
If patients receive a negative result they 
do not see the retina specialist  
No Intervention: Control group  
 

All participants see the retina specialist 
irrespective of the results of their 
autonomous AI evaluation.  
 
Outcome Measures 
Primary Outcome Measures: 
1. Number of examined participants with diabetes per retina specialist working hour 
a. Number of examined participants with diabetes per retina specialist 
working hour. Numerator is the number of examined participants 
(including persons evaluated by autonomous AI on Intervention Days who 
are determined not to need to see the retina specialist). The denominator 
is retina specialist working time in hours. 
2. Number of (all) patients per retina specialist working hour 
a. Number of examined patients (all types) per retina specialist working hour. 
Numerator is the number of examined participants (including persons 
evaluated by autonomous AI on Intervention Days who are determined not 
to need to see the retina specialist). The denominator is retina specialist 
working time in hours. 
 
Secondary Outcome Measures: 
1. Change in complexity score 
a. Change from baseline in mean complexity score of participants seen per 
hour per retina specialist. The complexity score is determined by external 
masked graders using a standard system adapted from Wilkinson et al. 
International Clinical Diabetic Retinopathy (DR) and Diabetic Macular 
Edema (DME) Severity Scales. The complexity score for an eye will be the 
sum of points with higher score as increased complexity (0=No or Mild 
Non-Proliferative DR (NPDR), 1 = Moderate NPDR or Severe NPDR, 3 = 
Proliferative DR (PDR), 2 = DME), and for a person it will be the sum of 
both eyes. 
2. Satisfaction with autonomous AI assessed by questionnaire. 

a. Retina specialist, technician and participant satisfaction with autonomous 
AI assessed by questionnaire using a 5 point Likert scale (1 = Very 
Satisfied 2 = Satisfied, 3 = Dissatisfied, 4 = Very Dissatisfied and 5 = N/A). 
 
Outline of main analysis: Raw and adjusted (for above physician and patient 
predictors of productivity) intra-group difference in change in the number of patients with 
diabetic retinopathy passing through the clinic per hour per physician, cases of DR 
treated per week, and complexity score for each ophthalmologist. 
 
Inclusion Criteria:  
Retina specialists regularly seeing patients with DR 
• Routinely examines >= 20 patients with diabetes without known diabetic 
retinopathy or diabetic macular edema per week 
• Routinely provides laser treatment or intravitreal injections to >= 3 DR 
patients/month 
AI Participants 
• Diagnosed with type 1 or 2 diabetes 
• >=22 years 
• Presenting visual acuity >= 6/18 best corrected visual acuity in the better-seeing 
eye 
 
Exclusion Criteria: 
Retina specialists: 
• Currently using an AI system integrated into their clinical care and/or inability to 
provide informed consent. 
AI Participants: 
• Inability to provide informed consent or understand the study; persistent vision 
loss, blurred vision or floaters; previously diagnosed with diabetic retinopathy or 
diabetic macular edema; history of laser treatment of the retina or injections into 
either eye, or any history of retinal surgery; contraindicated for imaging by fundus 
imaging systems 

B-PRODUCTIVE Statistical Analysis Plan 
1. Background and Design  
The purpose of the study is to assess the impact of using autonomous artificial 
intelligence (AI) (IDx-DR, Digital Diagnostics, USA) for identification of diabetic 
retinopathy (DR) and diabetic macular edema on productivity of retina specialists in 
Bangladesh. The Bangladesh PRODUCTIVity in Eyecare (B-PRODUCTIVE) Trial is a 
cluster-randomized (by clinic day) controlled trial. The trial will take place at a single 
clinic, Deep Eye Care Foundation (DECF), in Bangladesh collaborating with the on-
going Orbis World Diabetes Federation DR project.  
2. Outcome Measures 
2.1. 
Main Study Outcome: 
The number of examined patients with diabetes per retina specialist working hour 
between AI and non-AI days. 
• Numerator: number of examined patients  
o Sum of the number of patients diagnosed with diabetes mellitus (type 1 or 
type 2) examined for DR per day 
o Patients included have not been previously diagnosed with DR 
o Includes all patients evaluated by autonomous AI on AI days, even if they 
are not seen by the retina specialist  
• Denominator: retina specialist working time (hours)  
o Sum of retina specialist time spent on all diabetes patients examined per 
day 
o Does not include patient time spent during autonomous AI evaluation 
o Excluding treatment or surgical time  
o Including retina specialist time spent on ordering & reviewing diagnostic 
studies and scheduling treatment/surgery  
2.2 Secondary outcomes:  
2.2.1 Number of examined patients (all clinic patients, not only patients with 
diabetes) per retina specialist working hour between AI and non-AI days 
2.2.2 Number of patients with diabetic retinopathy per day scheduled for any 
treatment including injections, implants, laser or surgery. 
2.2.3 Retina specialist and AI-eligible patient satisfaction with autonomous AI.  
2.2.4 Mean complexity score of patients seen per hour per retina specialist. 
3. Data 
3.1. 
Data collection methods 
3.1.1. Ophthalmologist clinic records as audited by study personnel.  

3.1.2. Masked documentation of physician time spent with patients.  
3.1.3. Complexity score for each patient will be calculated by a masked UK NHS 
grader using the International Grading system, which is adapted from 
Wilkinson et al. International Clinical Diabetic Retinopathy and Diabetic 
Macular Edema Severity Scales.  
3.1.4. AI-eligible patient and Specialist questionnaires.  
3.2. 
Data Completion  
• Questionnaires will be completed by specialist participants at the end of the 
trial.   
• AI-eligible patient demographic data will be completed prior to the AI exam, 
and applicable satisfaction surveys will be completed following the AI exam.  
• Specialist hours will be tracked daily for clinic time and number of clinic 
exams completed by each specialist will be tracked daily (all patients 
examined and patients with diabetes examined) 
4. Sample Size Calculations  
The null hypothesis is that diabetic patients without known DR seen by retina specialist 
per doctor hour for regular clinic days and AI days are the same. We are estimating the 
sample size needed to reject this null hypothesis. In calculating the sample size we 
used 80% power (the probability of rejecting a false null hypothesis). The sample size 
estimate is 924 patient participants.  
Inputs for the sample size estimate calculations are: 
• 80% power; power is the probability of rejecting a false null hypothesis. 
• Alpha of 5%; probability of rejecting a true null hypothesis. 
• Average cluster size of 8 and cluster size coefficient of variation of 0.45 (based 
on March data).  
• Mean number of diabetic patients without known DR seen per doctor hour is 1.34 
(based on Quarter 1 data). Hospital records that were used to calculate the 
sample size show that there are approximately 366 diabetic patients seen at the 
hospital per month.  The retina clinic examines approximately 237 patients with 
diabetes per month.  
• We changed some factors, but decided on one estimate.  
Mean of 
Group 1 
Mean of 
Group 2 
Mean 
Difference 
Intra-cluster 
correlation 
coefficient 
Sample 
Size 
estimate 

1.34 
1.68 
0.34 
0.15 
924 
 
 
 
5. Randomization and Masking  
The Bangladesh PRODUCTIVity in Eyecare (B-PRODUCTIVE) Trial is a cluster-
randomized (by clinic day) controlled trial. The study will assess the change in number 
of examined patients with diabetes per retina specialist working hour for three fully-
qualified ophthalmologists regularly seeing patients with DR. All eligible patients (aged 
>= 22, known diabetes, no previous DR diagnosis, consented) receive autonomous AI 
for DR detection. This is done for purposes of masking (neither patients nor physicians 
know the group assignment of eligible patients) and concealment. On randomly 
selected “Intervention” clinic days, if eligible patients receive a positive DR diagnosis or 
have insufficient image quality, they continue to the ophthalmologist. If eligible patients 
receive a negative DR diagnosis, they do not see the retina specialist, and are referred 
for a visit at the regular eye clinic (not the retina clinic) in 3 months. On randomly 
selected “Control Days,” all patients see the ophthalmologist, irrespective of the results 
of the autonomous AI report. The retina doctors are masked both to patient group 
assignment (that is, whether autonomous AI was used or not on the particular clinic 
day) and also masked to the results of the autonomous AI on Intervention days. 
Patients are also masked to group assignment and autonomous AI results.   
6. Analysis  
Raw and adjusted (for above physician and patient predictors of productivity) group 
differences in the number of patients with diabetic retinopathy passing through the clinic 
per hour per physician will be calculated for AI and non-AI days. The two clinic day 
patient groups will compared with Chi-square testing for categorical variables. Age as a 
continuous variable was summarized with mean and standard deviation and the two 
groups will be compared with the student t-test.  
Study outcomes of encounters per retina specialist hour and confidence intervals will be 
calculated for each clinic day for all patients, as well as specifically for diabetic patients 
and retina specialty patients. Rates will be compared across AI and non-AI clinic days 
with student t-testing. Associations between the main outcome, diabetic patients 
examined per hour, and potential predictors will be assessed utilizing linear regression 
modeling with generalized estimating equations that include clustering effect of day. 
Residuals will be assessed to confirm that a linear model fits the rate outcome. A 
multivariable model will include sex, gender and variables that have p-value<0.10 in the 
univariate analysis. 

6.1 Recruitment and follow-up  
The flow of participants through each stage of screening, enrolment into the study, 
randomization and follow-up to study exit will be illustrated using a CONSORT flow 
diagram (Cluster extension).  
6.2 Baseline characteristics 
Clinic characteristics include number of clinic hours per day, total clinic hours during the 
trial, total completed specialist visits for diabetes patients and total completed specialist 
visits for all patients.  
AI-eligible patient characteristics include age, sex, and estimated monthly income.  
Specialist participant characteristics include age, sex, and years as a retina specialist.