<figure><img src="image_1.png" /></figure>

HAL
open science

RFMiD: Retinal Image Analysis for multi-Disease
Detection challenge

Samiksha Pachade, Prasanna Porwal, Manesh Kokare, Girish Deshmukh,
Vivek Sahasrabuddhe, Zhengbo Luo, Feng Han, Zitang Sun, Li Qihan,
Sei-Ichiro Kamata, et al.

<figure><img src="image_2.png" /></figure>

To cite this version:

Samiksha Pachade, Prasanna Porwal, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, et al.. RFMiD: Retinal Image Analysis for multi-Disease Detection challenge. Medical Image Analysis, 2025, 99, pp.103365. 10.1016/j.media.2024.103365. hal-05304065

HAL Id: hal-05304065

https://hal.science/hal-05304065v1

Submitted on 8 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

# RFMiD: Retinal Image Analysis for multi-Disease Detection Challenge

Samiksha Pachade<sup>a,1,*</sup>, Prasanna Porwal<sup>a,1</sup>, Manesh Kokare<sup>a,1</sup>, Girish Deshmukh<sup>b,1</sup>, Vivek Sahasrabuddhe<sup>c,1</sup>, Zhengbo Luo<sup>d</sup>, Feng Han<sup>e</sup>, Zitang Sun<sup>d</sup>, Li Qihan<sup>d</sup>, Sei-ichiro Kamata<sup>d</sup>, Edward Ho<sup>f</sup>, Edward Wang<sup>f</sup>, Asaanth Sivajohan<sup>f</sup>, Saerom Youn<sup>f</sup>, Kevin Lane<sup>f</sup>, Jin Chun<sup>f</sup>, Xinliang Wang<sup>g</sup>, Yunchao Gu<sup>g</sup>, Sixu Lu<sup>h</sup>, Young-tack Oh<sup>i</sup>, Hyunjin Park<sup>j,k</sup>, Chia-Yen Lee<sup>l</sup>, Hung Yeh<sup>m,1</sup>, Kai-Wen Cheng<sup>l</sup>, Haoyu Wang<sup>n</sup>, Jin Ye<sup>o</sup>, Junjun He<sup>n,o</sup>, Lixu Gu<sup>n</sup>, Dominik Müller<sup>P,q</sup>, Iñaki Soto-Rey<sup>P,q</sup>, Frank Kramer<sup>P</sup>, Hidehisa Arai<sup>r</sup>, Yuma Ochi<sup>s</sup>, Takami Okada<sup>t</sup>, Luca Giancardo<sup>u,1</sup>, Gwenolé Quellec<sup>v,1</sup>, Fabrice Mériaureau<sup>w,1</sup>

## ARTICLE INFO

**Article history:**

Received –

Received in final form –

Accepted –

Available online –

Communicated by –

**Keywords:** retinal fundus images, rare pathology detection, ocular disease, classification, multi-label classification

## ABSTRACT

In the last decades, many publicly available large fundus image datasets have been collected for diabetic retinopathy, glaucoma, and age-related macular degeneration, and a few other frequent pathologies. These publicly available datasets were used to develop a computer-aided disease diagnosis system by training deep learning models to detect these frequent pathologies. One challenge limiting the adoption of a such system by the ophthalmologist is, computer-aided disease diagnosis system ignores sight-threatening rare pathologies such as central retinal artery occlusion or anterior ischemic optic neuropathy and others that ophthalmologists currently detect. Aiming to advance the state-of-the-art in automatic ocular disease classification of frequent diseases along with the rare pathologies, a grand challenge on "Retinal Image Analysis for multi-Disease Detection" was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI - 2021). This paper, reports the challenge organization, dataset, top-performing participants solutions, evaluation measures, and results based on a new "Retinal Fundus Multi-disease Image Dataset" (RFMiD). There were two principal sub-challenges: disease screening (i.e. presence versus absence of pathology - a binary classification problem) and disease/pathology classification (a 28-class multi-label classification problem). It received a positive response from the scientific community with 74 submissions by individuals/teams that effectively entered in this challenge. The top-performing methodologies utilized a blend of data-preprocessing, data augmentation, pre-trained model, and model ensembling. This multi-disease (frequent and rare pathologies) detection will enable the development of generalizable models for screening the retina, unlike the previous efforts that focused on the detection of specific diseases.

<footnote><sup>a</sup>Center of Excellence in Signal and Image Processing, Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded 431606, India</footnote>

<footnote><sup>b</sup>Eye Clinic, Sushrusha Hospital, Nanded 431601, India</footnote>

<footnote><sup>c</sup>Department of Ophthalmology, Shankarrao Chavan Government Medical College, Nanded 431606, India</footnote>

<footnote><sup>d</sup>Graudate School of Information Production and Systems, Waseda University, Japan</footnote>

<footnote><sup>e</sup>University of Shanghai for Science and Technology, Shanghai, China</footnote>

<footnote><sup>f</sup>Schulich Applied Computing in Medicine, University of Western Ontario, Schulich School of Medicine and Dentistry</footnote>

<footnote><sup>g</sup>Beihang University School of computer science</footnote>

<footnote><sup>h</sup>Beijing Normal University School of artificial intelligence</footnote>

<footnote><sup>i</sup>Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea</footnote>

<footnote><sup>j</sup>Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Korea</footnote>

<footnote><sup>k</sup>School of Electronic and Electrical Engineering, Sungkyunkwan University, Suwon, Korea</footnote>

<footnote><sup>l</sup>Department of Electrical Engineering, National United University, Miaoli 360001, Taiwan(R.O.C)</footnote>

<footnote><sup>m</sup>Institute of Biomedical Engineering, National Yang Ming Chiao Tung University, 1001 Ta-Hsueh Road Hsinchu, Taiwan(R.O.C)</footnote>

<footnote><sup>n</sup>School of Biomedical Engineering, the Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China</footnote>

<footnote><sup>o</sup>Shenzhen Key Lab of Computer Vision and Pattern Recognition, Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China</footnote>

<footnote><sup>p</sup>IT-Infrastructure for Translational Medical Research, University of Augsburg, Germany</footnote>

<footnote><sup>q</sup>Medical Data Integration Center, University Hospital Augsburg, Germany</footnote>

<footnote><sup>r</sup>NABLAS Inc</footnote>

<footnote><sup>s</sup>National Institute of Technology, Kisarazu College, Japan</footnote>

<footnote><sup>t</sup>Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health</footnote>

<footnote><sup>u</sup>Center for Precision Health, School of Biomedical Informatics, University of Texas Health Science Center at Houston (UTHealth), Houston, TX 77030, USA</footnote>

<footnote><sup>v</sup>Inserm, UMR 1101, F-29200 Brest, France</footnote>

<footnote><sup>w</sup>ImViA EA 7535/IFTIM, Université de Bourgogne, 21078 Dijon, France</footnote>
# 1. Introduction

Eye conditions are prevalent, and on the rise. According to the world health organization (WHO), world report on vision 2019, at present, the total number of people around the world with vision impairment is estimated to be 2.2 billion, from which at least 1 billion vision impairment cases could have been prevented or are yet to be addressed. These over 1 billion people are living with treatable or preventable conditions, just because they can't get the eye care they need. The global demand for eye care is set to swell in the coming years because of rapid population growth, aging, and lifestyle changes. Vision impairment also causes an immense global financial burden due to productivity loss (WHO, 2019).

Color fundus retinal photography uses a camera with a specialized low-powered microscope to non-invasively capture the interior surface of the eye. Retinal fundus images are widely used for clinical screening, examining the presence of disorders, and monitor their change over time. The world faces considerable challenges in terms of eye care, including inequalities in the coverage and quality of prevention, treatment, and rehabilitation services. Detailed clinical observations of characteristic retinal fundus features—not only provide information about eye diseases—but also have led to the identification of early indicators of a diverse range of long-term conditions, such as diabetes, stroke, hypertension, arteriosclerosis, cardiovascular, neurodegenerative, and renal and fatty liver diseases (MacGillivray et al., 2014; Yang et al., 2015; Chang et al., 2016). Hence, the retina is a potential surrogate organ whose assessments are significant not just for the health of the eye but also to decide whether there are systemic issues. Thus, vision loss and damage to other parts of the body can be avoided by eye screening coupled with timely detection and treatment.

Advances in automated retinal image analysis can be used for early detection and diagnosis of ocular pathologies, which would enable the forestall of visual impairment, reduce workload associated with manual grading by ophthalmologists and optometrists, and save the cost of diagnosis and time. As the ratio of ophthalmologists and optometrists to the population is approximately 1:10,000 in most developed countries. However, the ratio is 1:600,000 in developing countries and millions of people per ophthalmologist in rural areas (OAM and Resnikoff, 2002). This inadequacy of professionals is the primary reason for high rates of vision issues in developing countries. Thus, switching to automated computerized screening systems will help in the detection of main components of the retina, classify the normal, abnormal, and unknown retinal images, which in turn significantly increase the rates of disease detection and treatment in the initial stage of its progression.

Recent scientific and technological advances in computing capacity and artificial intelligence approaches provide a new ability for biomedical scientists to meet the need of clinical

practice (Shortliffe and Blois, 2006; Patton et al., 2006). To meet this requirement, raw retinal images along with their associated ground truths play an important role by providing a common ground for the development, validation, and comparison of techniques to the research community. For multi-disease detection, the ground truths are labels given to the particular image based on the clinical records and visual fields. This need has driven the development and sharing of the retinal image databases by the research groups. In the last few decades, some datasets representing diabetic patients collected through diabetic retinopathy (DR) screening programs are IDRiD (Porwal et al., 2020, 2018a), Messidor (Decencière et al., 2014), and EyePACS (Cuadros and Bresnick, 2009). Also, datasets representing the sight-threatening pathologies such as Glaucoma are Drishti-GS (Sivaswamy et al., 2015), RIM-ONE (Fumero et al., 2011), and Refuge (Orlando et al., 2020), and representing age-related macular degeneration (AMD) are ARIA (Farnell et al., 2008), STARE (Hoover et al., 2000), and (Group et al., 2001). The patients visiting the DR screening program may also suffer from other diseases, such as Glaucoma and AMD; thus, multi-disease detection is crucial. There are few datasets that can be used for multi-disease detection one such dataset is ODIR challenge ODIR-2019 (2019) which is publicly available. In ODIR dataset detailed anomaly labeling is only at patient level, not eye level. Also, it only focused on 8 labels (most anomalies being grouped in the 'other' category for evaluation). These multi-disease detection datasets are either private or incorporate only the most common pathologies. However, detection and diagnosis of other sight-threatening rare pathologies, like central retinal artery occlusion or anterior ischemic optic neuropathy and others is also essential. Due to the unavailability of datasets representing rare pathologies, the methodologies for automated detection of rare pathologies are not developed. Thus, rare pathologies are ignored by the automated screening system that usually occurs in routine clinical checkups. This restricts the adoption of automated screening systems by ophthalmologists.

To address these issues, we introduced a new dataset called “Retinal Fundus Multi-Disease Image Dataset” (RFMiD) (Pachade et al., 2021b). Further, it was used as a base dataset for the organization of the grand challenge “Retinal Image Analysis for multi-Disease Detection” (RIADD) in conjunction with International Symposium on Biomedical Imaging (ISBI - 2021). The RFMiD dataset provides 3200 color fundus images with image-level labels representing a normal or abnormal class. It also provides labels for 45 different diseases/abnormalities annotated by the retinal experts. This challenge brought biomedical and computer vision researchers together, intending to develop generalizable techniques/networks for screening retina, unlike the previous efforts that focused on the detection/classification of specific diseases. Initially, for the development of the algorithm, participants were provided with a training dataset and the ground truth. Later, the individuals/teams to participate in the final phase of the challenge were shortlisted based on the performance of their algorithm on the validation dataset. Finally, the winners were judged based on the test dataset performance. The performance of the algorithm

<footnote>*Corresponding author
e-mail: 2017pec601@sggs.ac.in (Samiksha Pachade)</footnote>

<footnote n="1">¹These authors co-organized the challenge. All others contributed results of their algorithm(s) presented in the paper</footnote>
was estimated by how closely the results of the algorithm match the ground truth. The challenge was divided into two sub-tasks: disease screening and disease/pathology classification.

This challenge paper offers unique contributions. Here are key points highlighting its contributions:

1. Focus on Rare Pathologies: Our challenge uniquely addresses the underdeveloped methodologies for automated detection of rare pathologies by providing datasets representing a broader spectrum of conditions, filling a crucial gap in the literature.

2. Enhancing Generalizability: Our challenge promotes solutions applicable to a broader range of conditions, fostering the development of versatile and robust screening systems, in contrast to previous efforts focusing on specific diseases.

3. Addressing Data Challenges: The challenge tackles issues related to the problem of multi-disease detection, limited data, and class imbalance. It provides insights into handling these challenges through a substantial number of practical tips, contributing valuable knowledge for researchers and practitioners in the field.

4. Development of New Solutions: The challenge has led to the development of a variety of new solutions for multi-disease detection. This outcome highlights the effectiveness of the collaborative efforts and the significance of the challenge in advancing the state-of-the-art in the field.

The rest of the paper is organized as follows: section 2 provides a short review of retinal image analysis for multi-disease detection, section 3 gives details of the RFMiD dataset, section 4 provides the organization of the competition, section 5 describes the summary of top-performing participants solutions, section 6 describes evaluation measures used for performance evaluation in this challenge, and section 7 presents the result and comparison of participating teams for both sub-challenges, section 8 gives a brief discussion and conclusion.

## 2. Review of Challenges and Retinal Image Analysis for Multi-disease Detection

In the past decade, few challenges are organized for retinal image analysis, for DR (IDRiD Porwal et al. (2018a), DRIVE Staal et al. (2004), ROC Niemeijer et al. (2009), Kaggle DR detection Cuadros and Bresnick (2009), DeepDRiD Liu et al. (2022)), Glaucoma (REFUGE Orlando et al. (2020), AIROGS de Vente et al. (2021)), AMD (ADAM Fu et al. (2020)), pathological myopia (PM) (PALM PALM-2019 (2019)), and multipathology (ODIR ODIR-2019 (2019)). Such challenges are well known to enable advances in the field by promoting participation of scientific research community from all over the world in a competition at the same time for scientific advancement. IDRiD challenge Porwal et al. (2020, 2018a) dataset consists of 516 retinal images with information on disease severity of DR, diabetic macular edema, and normal retinal structures (optic disc and fovea center coordinates) for each image. Also, it provides 81 images with typical DR lesions (microaneurysms, hemorrhages, hard exudates, and soft exudates) and

optic disc annotated at a pixel level. DRIVE Challenge Staal et al. (2004) dataset consists of 40 images with pixel-level vessel ground truth for all images marked by three experts. ROC Challenge Niemeijer et al. (2009) dataset consists of 100 fundus images for detection of microaneurysms (primary signs of DR) which are marked by four experts. Kaggle DR detection Challenge Cuadros and Bresnick (2009) dataset consists of 88,702 retinal color fundus images. The dataset provides severity grading for DR labeled by two experts. DeepDRiD challenge Liu et al. (2022) dataset composed of 2,000 regular DR images and 256 ultra-widefield images, both having DR quality and grading annotations. REFUGE Challenge Orlando et al. (2020) dataset consists of total 1200 retinal fundus images. The dataset provides ground truth for OD and OC segmentation, fovea localization, and classification of clinical glaucoma annotated by seven independent retinal experts. AIROGS challenge de Vente et al. (2021) dataset contains 113,893 color fundus images with the labels referable glaucoma, no referable glaucoma, or ungradable. ADAM challenge Fu et al. (2020) dataset consists of 1200 images with annotations for diagnosis of AMD, detection and segmentation of optic disc, fovea localization, and segmentation of lesions in fundus photos from AMD patients. The images are annotated by seven independent retinal experts. PALM Challenge PALM-2019 (2019) dataset consists of 1200 images with ground truth for OD segmentation and localization, fovea localization, lesion (atrophy and detachment) segmentation, and classification of PM and non-PM fundus images annotated by seven independent experts. ODIR challenge ODIR-2019 (2019) dataset consists of 5000 images. The images were given 8 different labels namely, normal, diabetes, glaucoma, cataract, AMD, hypertension, myopia, and other diseases/abnormalities.

Many previous studies have focused on retinal disease detection and image analysis using machine learning approaches. Several machine learning algorithms such as the k-nearest neighbor (kNN) algorithm, artificial neural network (ANN), Naive Bayes classifier, and support vector machine (SVM) have been used for retinal disease detection (Mookiah et al., 2013). For retinopathy lesions classification SVM, Gaussian mixture model (GMM), kNN, and AdaBoost are utilized (Roychowdhury et al., 2013).

With the recent success of deep learning, numerous automatic screening systems for fundus images were recently proposed. Historically most targeted pathologies by those systems are DR, AMD, and glaucoma. This is because large datasets of fundus images representing these diseases have been collected over the past decades. Automatic screening system were proposed for DR (Abràmoff et al. (2016), Gulshan et al. (2016), Raju et al. (2017), Gargeya and Leng (2017), Mansour (2018), Nielsen et al. (2019), Quellec et al. (2019)), glaucoma (Li et al. (2018), Shibata et al. (2018), Li et al. (2018), Christopher et al. (2018), Raghavendra et al. (2018), Ahn et al. (2018), Phan et al. (2019), Diaz-Pinto et al. (2019), Pachade et al. (2021a)), and AMD (Grassmann et al. (2018), Tan et al. (2018), Pead et al. (2019), Peng et al. (2020)). Automated frameworks for the detection of other pathologies such as retinopathy of prematurity (Wang et al. (2018), Hu et al. (2018a)) were also developed.
In addition to DR detection Porwal et al. (2018b) proposed a technique for AMD detection. Similarly, Ting et al. (2017) proposed a method for AMD and glaucoma detection along with DR detection.

Recent studies highlight the importance of efforts in the direction of multi-disease detection. Choi et al. (2017) applied a convolutional neural network (CNN) on fundus images for multi-categorical disease detection. VGG-19 architecture with random forest transfer learning gave the most favorable outcomes. The performance of the deep learning models were diminished with the increase in the number of categories. Keel et al. (2019) presented a heat map visualization network using CNNs for the detection of referable DR and glaucomatous optic neuropathy. Son et al. (2020) presented deep learning algorithms that detect the presence of multiple abnormal findings and show lesion heat maps of multiple abnormal findings in retinal fundus photographs. Quellec et al. (2020) first time presented the automatic detection of a large number of (frequent and rare) conditions in retinal images. A new few-shot learning framework was proposed for this purpose. Initially, for the detection of frequent conditions, a deep learning classifier is trained and then for rare condition detection, unsupervised probabilistic models are obtained from these deep learning models.

# 3. Retinal Fundus Multi-disease Image Dataset

## 3.1. Data Acquisition

The retinal images were acquired using three different digital fundus cameras, TOPCON 3D OCT-2000, Kowa VX-10α, and TOPCON TRC-NW300 during the period 2009–2020. They are centered either on the macula or the optic disc (OD). These images are acquired at an Eye Clinic, Sushrusha Hospital, and Center of Excellence in Signal and Image Processing, SGGS Institute of Engineering and Technology both located in Nanded, (M.S.), India. The fundus images are collected from subjects visiting due to a concern for their eye health. Prior to image capturing, the pupils of most of the patients were subjected to mydriasis with one drop of tropicamide at 0.5% concentration. Fundus image acquisition is done with the position and orientation of the subjects sitting upright with 39 mm (Kowa VX-10) and 40.7 mm (TOPCON 3D OCT-2000 and TOPCON TRC-NW300) distance between lenses and examined eye. The details of the camera model, field of view (FOV), resolution, and number of images included in the dataset are given in table 1. The dataset provides expert disease annotation.

<table><caption>Table 1. The details of the camera model, field of view (FOV), resolution, and number of images included in the dataset.</caption><thead><tr><td>Model</td><td>FOV</td><td>Resolution (in pixels)</td><td>Number of images in dataset</td></tr></thead><tbody><tr><td>TOPCON 3D OCT - 2000</td><td>45°</td><td>2144 × 1424</td><td>2427</td></tr><tr><td>Kowa VX - 10α</td><td>50°</td><td>4288 × 2848</td><td>467</td></tr><tr><td>TOPCON TRC-NW300</td><td>45°</td><td>2048 × 1536</td><td>306</td></tr></tbody></table>

## 3.2. Annotation of Images

It comprises information that represents the overall risk factor based on the assessment of the entire image. Two ophthalmologists independently graded the full set of 3200 fundus images. They used a thorough assessment of patients' clinical records and visual fields as the reference standard for the presence of various diseases. The images depict the presence of no disease, single disease, or more than one disease. If it shows the presence of more than one disease, then multiple labels are assigned to the single image. After completion of the initial labeling by both the ophthalmologists, the leader of the project team checked and confirmed or corrected through consultation from both the ophthalmologists when the difference in diagnostic assessments was observed to get adjudicated consensus for the labels. Ground truth for all the fundus images is available in the CSV file.

A brief explanation of the disease/abnormality with their significant visual characteristics of all 46 categories (1 normal and 45 diseases/pathologies) is given in our data descriptor paper (Pachade et al., 2021b). (https://doi.org/10.3390/data6020014).

# 4. Challenge Organization

To boost the efforts towards the development of a generalizable retinal screening system, this dataset is available as a part of RIADD challenge organized in conjunction with IEEE ISBI-2021, Nice, France.

The RIADD challenge was composed of various stages. Figure 1 shows the overall challenge organization workflow. The challenge was officially announced on 16th October 2020 at ISBI-2021 website (http://biomedicalimaging.org/2021/challenges/). The challenge is divided into two sub-challenges as follows:

1. Screening of retinal images into normal and abnormal (comprising of 45 different types of diseases/pathologies) categories.

2. Classification of retinal images into 28 different categories.

For sub-challenge 2, the diseases having more than 10 images belong to an independent class and all other disease categories were merged and labeled as “OTHER”. This finally constitute 28 diseases/pathologies groups for disease classification.

The challenge is comprised of 4 stages, given as follows:

## 4.1. Stage 1: Data Preparation and Distribution

The RFMiD dataset is formed by extracting 3200 images from the thousands of examinations. Both high-quality and low-quality images are selected to make the dataset challenging. The dataset is split into 3 subsets for training 60% (1920 images), evaluation set 20% (640 images), and test set 20% (640 images). The disease-wise stratification on average in training, evaluation, and test set is 60% ± 7%, 20% ± 7% and 20% ± 5% respectively. The detailed dataset breakdown is shown in table 2.
<figure><img src="image_6.png" /><figcaption>Fig. 1. Workflow of the ISBI - 2021: Retinal Image Analysis for multi-Disease Detection (RIADD) Challenge</figcaption></figure>

<table><caption>Table 2. Summary of data augmentation, normalization and pre-processing in the competing solutions. Where, RF, RR, RS, RT, RC represent random flip, rotation, scaling, translation and crop respectively.</caption><thead><tr><th>Sr. No.</th><th>Normal/Disease/Marker</th><th># of Training Set Images</th><th># of Validation Set Images</th><th># of Test Set Images</th><th>Sr. No.</th><th>Normal/Disease/Marker</th><th># of Training Set Images</th><th># of Validation Set Images</th><th># of Test Set Images</th></tr></thead><tbody><tr><td>1</td><td>NL</td><td>401</td><td>134</td><td>134</td><td>24</td><td>CRS</td><td>32</td><td>11</td><td>11</td></tr><tr><td>2</td><td>DR</td><td>376</td><td>132</td><td>134</td><td>25</td><td>EDN</td><td>15</td><td>5</td><td>4</td></tr><tr><td>3</td><td>ARMD</td><td>100</td><td>38</td><td>31</td><td>26</td><td>RPEC</td><td>22</td><td>6</td><td>4</td></tr><tr><td>4</td><td>MH</td><td>317</td><td>102</td><td>104</td><td>27</td><td>MHL</td><td>11</td><td>3</td><td>3</td></tr><tr><td>5</td><td>DN</td><td>138</td><td>46</td><td>46</td><td>28</td><td>RP</td><td>6</td><td>2</td><td>2</td></tr><tr><td>6</td><td>MYA</td><td>101</td><td>34</td><td>32</td><td>29</td><td>CWS</td><td>3</td><td>3</td><td>2</td></tr><tr><td>7</td><td>BRVO</td><td>73</td><td>23</td><td>23</td><td>30</td><td>CB</td><td>1</td><td>0</td><td>1</td></tr><tr><td>8</td><td>TSLN</td><td>186</td><td>65</td><td>53</td><td>31</td><td>ODPM</td><td>0</td><td>2</td><td>0</td></tr><tr><td>9</td><td>ERM</td><td>14</td><td>7</td><td>5</td><td>32</td><td>PRH</td><td>2</td><td>1</td><td>2</td></tr><tr><td>10</td><td>LS</td><td>47</td><td>17</td><td>15</td><td>33</td><td>MNF</td><td>3</td><td>0</td><td>0</td></tr><tr><td>11</td><td>MS</td><td>15</td><td>5</td><td>7</td><td>34</td><td>HR</td><td>0</td><td>0</td><td>1</td></tr><tr><td>12</td><td>CSR</td><td>37</td><td>11</td><td>13</td><td>35</td><td>CRAO</td><td>2</td><td>2</td><td>0</td></tr><tr><td>13</td><td>ODC</td><td>282</td><td>72</td><td>91</td><td>36</td><td>TD</td><td>3</td><td>2</td><td>4</td></tr><tr><td>14</td><td>CRVO</td><td>28</td><td>8</td><td>9</td><td>37</td><td>CME</td><td>4</td><td>1</td><td>2</td></tr><tr><td>15</td><td>TV</td><td>6</td><td>2</td><td>2</td><td>38</td><td>PTCR</td><td>5</td><td>1</td><td>0</td></tr><tr><td>16</td><td>AH</td><td>16</td><td>4</td><td>5</td><td>39</td><td>CF</td><td>3</td><td>2</td><td>1</td></tr><tr><td>17</td><td>ODP</td><td>65</td><td>26</td><td>24</td><td>40</td><td>VH</td><td>1</td><td>2</td><td>1</td></tr><tr><td>18</td><td>ODE</td><td>58</td><td>21</td><td>17</td><td>41</td><td>MCA</td><td>1</td><td>0</td><td>0</td></tr><tr><td>19</td><td>ST</td><td>5</td><td>4</td><td>2</td><td>42</td><td>VS</td><td>1</td><td>3</td><td>0</td></tr><tr><td>20</td><td>AION</td><td>17</td><td>5</td><td>4</td><td>43</td><td>BRAO</td><td>2</td><td>2</td><td>0</td></tr><tr><td>21</td><td>PT</td><td>11</td><td>2</td><td>4</td><td>44</td><td>PLQ</td><td>1</td><td>1</td><td>0</td></tr><tr><td>22</td><td>RT</td><td>14</td><td>6</td><td>5</td><td>45</td><td>HPED</td><td>1</td><td>0</td><td>0</td></tr><tr><td>23</td><td>RS</td><td>43</td><td>14</td><td>14</td><td>46</td><td>CL</td><td>1</td><td>0</td><td>1</td></tr></tbody></table>

NL: normal, DR: diabetic retinopathy, ARMD: age-related macular degeneration, MZ: media haze, DN: drusen, MYA: myopia, BRVO: branch retinal vein occlusion, TSLN: tessellation, ERM: epiretinal membrane, LS: laser scar, MS: macular scar, CSR: central serous retinopathy, ODC: optic disc cupping, CRVO: central retinal vein occlusion, TV: tortuous vessels, AH: asteroid hyalosis, ODP: optic disc pallor, ODE: optic disc edema, ST: shunt, AION: anterior ischemic optic neuropathy, PT: parafoveal telangiectasia, RT: retinal traction, RS: retinitis, CRS: chorioretinitis, EDN: exudation, RPEC: retinal pigment epithelium changes, MHL: macular hole, RP: retinitis pigmentosa, CWS: cotton wool spots, CB: coloboma, ODPM: optic disc pit maculopathy, PRH: preretinal hemorrhage, MNF: myelinated nerve fibers, HR: hemorrhagic retinopathy, CRAO: central retinal artery occlusion, TD: tilted disc, CME: cystoid macular edema, PTCR: post traumatic choroidal rupture, CF: choroidal folds, VH: vitreous hemorrhage, MCA: macroaneurysm, VS: vasculitis, BRAO: branch retinal artery occlusion, PLQ: plaque, HPED: hemorrhagic pigment epithelial detachment, CL: collateral

The challenge was hosted on Grand Challenges in Biomedical Imaging Platform (https://grand-challenge.org/), one of the popular platforms for biomedical imaging-related competitions. A challenge website was launched on 1st November 2020 to disseminate challenge-related information. It was also used for registration, data distribution, submission of results and papers, and communication between the organizers and participants.

## 4.2. Stage 2: Registration and Training Data Release

The challenge registration for consideration to ISBI-2021 on-site contest was made open from 8th November 2020 to 28th November 2021.

February 2021. Individual researchers and research teams registered through the challenge website. The training dataset (images and ground truth) was made available on 22nd November 2020. Participants downloaded the dataset and developed or modified their methods. Further, the participants were allowed to use other datasets for the development of a method that was submitted to the challenge, provided that the datasets are publicly available and clearly stated in their submitted paper. Also, the teams were allowed to do private annotations on public data for some specific tasks and then train their models. If they do so, teams need to release their data and annotations to the public through the challenge website before 31st December 2020.
Only one team used other publicly available dataset for training the network without any annotations i.e. in semi-supervised manner. Thus, no data and annotations were released by any team.

### 4.3. Stage 3: Validation Data Release

The validation dataset (only images) was made available on 29th November 2020. Submissions were sought for either of the two different tasks. The result evaluation site was made open for submission from 1st December 2020. All participants were allowed to make up to 5 result submissions per day. The participants could submit their results by 28th February 2021 and a paper describing the approach till 5th March 2021 to the organizers. Results for both sub-challenge were asked to be submitted in a single CSV file, if participants choose to participate in only one sub-challenge, then they are expected to fill other fields with zeros. A system for automatic evaluation of results was set and the performance is displayed on the evaluation leaderboard on the challenge website. The final rank on the leaderboard is determined by using performance in both sub-challenges as follows:

$$ \text{Final Score} = 0.5*(\text{Sub-challenge-1 Score}) + 0.5*(\text{Sub-challenge-2 Score}) $$

The 8 top-performing participating teams and individuals (decided by the experts based on the performance on the evaluation dataset and their method) were invited to participate in the on-site challenge.

### 4.4. Stage 4: Test Data Release and ISBI-2021 Challenge Event

The main challenge event was held in conjunction with the IEEE ISBI-2021, on 13th April 2021 through a virtual platform. The test dataset was made available on the challenge website (online mode). The participants were given one hour to work on the test data and submit their results. Further, the top three teams from both sub-challenges presented their approach. Finally, the summary of the challenge and final ranking of the competing solutions were presented.

## 5. Summary of Competing Solutions

In this section, we discuss the competition solutions of top performing 8 teams in terms of the following four sections: data pre-processing, data augmentation, pre-trained model, and deep learning models used for classification. The details of competing solutions of each teams are given in Appendix A.

### 5.1. Data Pre-processing

The outcome of any algorithm to solve any particular problem is heavily dependent on the proper distributed data. Thus, data pre-processing is a required first step in designing the network. We analyzed different pre-processing steps used by different teams for the disease screening and classification tasks. One of the pre-processing steps, region of interest (ROI) extraction method that cuts out the redundant black edge from fundus images was used by all 8 teams. No team adopted any background subtraction approach. A summary of the pre-processing steps of all the teams is given in table 3.

<table><caption>Table 3. Summary of data pre-processing used by all the top performing teams. Where ROI is extraction which cuts out the redundant black edge from fundus images; Resize is if the images are resized before giving to the network; CLAHE is if contrast limited adaptive histogram equalization (CLAHE) is applied to images; Green is if the green channel is extracted and given to the network.</caption><thead><tr><th>Team Name</th><th>ROI</th><th>Resize</th><th>CLAHE</th><th>Green</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>Schulich Applied Computing inMed</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>BNUAA</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>Nekaravuru</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>WWW</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>IGSTfencing</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>MISIT</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>chizu & arai & okada</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr></tbody></table>

### 5.2. Data Augmentation and Normalization

Data augmentation is the set of techniques that are used to artificially increase the size of the data by creating modified copies of images from the already existing dataset, which in turn prevents data scarcity. It is used to increase the generalizability of the data model by reducing data overfitting. It also helps in resolving the class imbalance issue in the classification. Most teams used data augmentation methods, such as flipping, rotation, cropping, and so on, to maintain the models generality. Along with these general augmentations, additional color adjustment methods, i.e., Cutout (DeVries and Taylor, 2017) and Mixup (Zhang et al., 2017) are used. While training the network data normalization makes the model converge faster. As most of the teams have used ImageNet pre-trained model, ImageNet normalization for each of the three RGB channel using mean = [0.485, 0.456, 0.406] and standard deviations = [0.229, 0.224, 0.225]. These values are calculated using millions of images from the ImageNet dataset. The summary of the data augmentation and whether ImageNet normalization is used for all the teams is given in table 4.

### 5.3. Deep Learning Models

Using a pre-trained model requires less effort in building the model's architecture, requires less training, and has proven to be accurate with proper fine-tuning as compared to a model using a custom-built CNN, especially when the number of annotated samples are quite low. In this challenge, the ImageNet pre-training model was used by most teams to improve their model performance. A summary of the pre-training model is given in table 5.

In this challenge, all teams used deep learning models to construct classification frameworks. Most teams adopted Efficient-Net (Tan and Le, 2019) as their deep learning backbones, and achieved excellent classification performance, whereas some teams selected classical ResNet (He et al., 2016), ResNext (Xie et al., 2017), InceptionV3 (Szegedy et al., 2016), DenseNet (Huang et al., 2017), SE-ResNeXt (Hu et al., 2018b), TRes-Net (Ridnik et al., 2021), SENet (Hu et al., 2018c) and NFNet (Brock et al., 2021). Team IGSTfencing used an external public dataset without labels for training the semi-supervised model. Most teams chose binary cross-entropy (BCE) loss function. All the teams used ensemble models which combine multiple
<table><caption>Table 4. Summary of data augmentation and normalization used by all the top performing teams. Where RF, RR, HSV, RB/RBC, RC, RRC, and SSR represents random flip, random rotation, hue saturation value, random brightness/ random brightness contrast, random crop, random resized crop, and shift scale rotate respectively.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="10">Augmentation</th><th rowspan="2">ImageNet normalization</th></tr><tr><th>RF</th><th>RR</th><th>HSV</th><th>RB/RBC</th><th>RC</th><th>RRC</th><th>SSR</th><th>Cutout</th><th>Mixup</th><th>Other</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>median blur, additive Gaussian noise</td><td>✓</td></tr><tr><td>Schulich Applied Computing inMed</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✗</td></tr><tr><td>BNUAA</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td><td>✓</td><td>resize, transpose, coarse dropout, random scale, shifting, blur, elastic transformations, grid distortion</td><td>✗</td></tr><tr><td>Nekaravuru</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>–</td><td>✗</td></tr><tr><td>WWW</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✗</td></tr><tr><td>IGSTfencing</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✓</td></tr><tr><td>MISIT</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✓</td></tr><tr><td>chizu & arai & okada</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>–</td><td>✗</td></tr></tbody></table>

<table><caption>Table 5. Summary of pre-training model used and deep learning model with loss functions and model ensembling. Where BCE is binary cross-entropy, CE is cross-entropy, SC1 is sub-challenge 1, and SC2 is sub-challenge 2.</caption><thead><tr><th>Team Name</th><th>ImageNet pre-trained mode</th><th>Model framework</th><th>Loss function</th><th>Model ensembling</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>✓</td><td>EfficientNet B5 and B6<br>SE-ResNeXt, InceptionV3,<br>EfficientNet B4 and B5,<br>and DenseNet-121</td><td>BCE with logits</td><td>✓</td></tr><tr><td>Schulich Applied Computing inMed</td><td>✓</td><td>ResNet and ResNext</td><td>asymmetric loss</td><td>✓</td></tr><tr><td>BNUAA</td><td>✗</td><td>EfficientNet B0, B1, and B2</td><td>CE and BCE loss</td><td>✓</td></tr><tr><td>Nekaravuru</td><td>✓</td><td>EfficientNet and TResNet</td><td>weighted BCE loss</td><td>✓</td></tr><tr><td>WWW</td><td>✓</td><td>SS-GCN with SENet154<br>as backbone</td><td>asymmetric loss</td><td>✓</td></tr><tr><td>IGSTfencing</td><td>✓</td><td>SC1: EfficientNet B4 and InceptionV3<br>SC2: ResNet152, InceptionV3,<br>DenseNet-201, and EfficientNet B4</td><td>BCE, multi-view<br>and co-regularization</td><td>✓</td></tr><tr><td>MISIT</td><td>✓</td><td>EfficientNet and NFNet</td><td>weighted focal loss</td><td>✓</td></tr><tr><td>chizu & arai & okada</td><td>✓</td><td></td><td>focal loss</td><td>✓</td></tr></tbody></table>

models in the prediction process. Specifically, the majority of teams opted for the averaging ensembling technique. This approach involves averaging predictions from individual models to derive the final prediction. Notably, team MISIT employed a combination of bagging and stacking ensembling techniques, while team Chizu & Arai & Okada utilized a weighted averaging ensembling technique. Bagging involves training multiple instances of the same base model on distinct subsets of the training data. The predictions from each model in the bagging ensemble are then aggregated through stacking. It's important to highlight that all teams generated multiple predictions using different models, rather than relying on a single model with augmented versions of the same inference image. This ensemble of diverse models significantly contributes to the overall robustness and accuracy of the predictions across all teams. A summary of the deep learning model with loss and model ensemble is given in table 5.

Sub-challenge-1 (disease screening) comprises two categories: normal and abnormal. Sub-challenge-2 (disease classification) comprises 28 different categories. The disease risk label in sub-challenge-1 aligns consistently with multiple labels representing different diseases and pathologies in sub-challenge-2. This indicates that the labels across multiple categories contain sufficient information relevant to both sub-challenge-1 and sub-challenge-2. Thus, the majority of teams approach combined handling of both sub-challenges as a unified task with 29 classes (1 class for sub-challenge-1 and 28 classes for sub-challenge-2). Teams BNUAA and WWW employed a single deep learning model as the base model for addressing both sub-challenges. Furthermore, the data pre-

processing, augmentation techniques, and training details utilized by all the teams are consistent for both sub-challenges.

## 6. Evaluation Measures

For the RIADD challenge, model performance was assessed with two metrics, “Disease Risk Score” and “Multi-Disease Average Score” each corresponding to one of the two sub-challenges: 1. Disease Screening: the ability of the model to detect the presence of any disease. This was measured by the (“Disease Risk Score”) AUC for the predictions of the normal/abnormal label only. 2. Disease Classification: the performance of the model in labelling a given image with the correct pathological findings. This was measured by (“Multi-Disease Average Score”) averaging the mean average precision (mAP) over each label with the average AUC over each label:

$$ \text{Multi-disease Avg Score} = \frac{1}{2} \left( \frac{1}{c} \sum_{i=0}^{c} AUC_{i} + mAP_{i} \right) \quad (1) $$

where c is the class index, and mAP is defined as:

$$ mAP = \sum_{n}^{N} (R_{n} - R_{n-1})p_{n} \quad (2) $$

where N is the number of unique threshold values, and $p_n$ and $R_n$ are the precision and recall, respectively, at the n-th threshold.

Additionally, we conducted an analysis for both tasks, evaluating F1 score and Specificity. It's noteworthy that, during
<table><caption>Table 6. The best submission performance of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi-Disease Average Score) on validation set. AUC: area under the receiver operator curve, mAP: mean average precision, SC1: sub-challenge-1 and SC2: sub-challenge-2. Average values are reported for SC2.</caption><thead><tr><th rowspan="2">Team Name</th><th rowspan="2">Disease Risk Score AUC</th><th rowspan="2">Multi-Disease Average Score AUC + mAP</th><th rowspan="2">Final Score</th><th rowspan="2">Rank</th><th colspan="2">SC1</th><th colspan="2">SC2</th></tr><tr><th>F1 Score</th><th>Specificity</th><th>F1 Score</th><th>Specificity</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9702</td><td>0.8017</td><td>0.8859</td><td>1</td><td>0.9411</td><td>0.8783</td><td>0.5561</td><td>0.9935</td></tr><tr><td>IGSTfencing</td><td>0.9601</td><td>0.7911</td><td>0.8756</td><td>2</td><td>0.9409</td><td>0.7612</td><td>0.5433</td><td>0.9917</td></tr><tr><td>chizu & arai & okada</td><td>0.9636</td><td>0.7873</td><td>0.8755</td><td>3</td><td>0.9382</td><td>0.8308</td><td>0.6247</td><td>0.9911</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9613</td><td>0.7858</td><td>0.8735</td><td>4</td><td>0.9335</td><td>0.5896</td><td>0.5467</td><td>0.9847</td></tr><tr><td>BNUAA</td><td>0.9610</td><td>0.7820</td><td>0.8715</td><td>5</td><td>0.9439</td><td>0.8433</td><td>0.5904</td><td>0.9931</td></tr><tr><td>Nekaravuru</td><td>0.9626</td><td>0.7566</td><td>0.8596</td><td>6</td><td>0.9369</td><td>0.8806</td><td>0.5013</td><td>0.9846</td></tr><tr><td>WWW</td><td>0.9572</td><td>0.7584</td><td>0.8578</td><td>7</td><td>0.9339</td><td>0.8060</td><td>0.5213</td><td>0.9886</td></tr><tr><td>MISIT</td><td>0.9490</td><td>0.7027</td><td>0.8258</td><td>8</td><td>0.9261</td><td>0.8731</td><td>0.6614</td><td>0.9900</td></tr></tbody></table>

<table><caption>Table 7. The performance of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi-Disease Average Score) on test set. AUC: area under the receiver operator curve, mAP: mean average precision, SC1: sub-challenge-1 and SC2: sub-challenge-2. Average values are reported for SC2.</caption><thead><tr><th rowspan="2">Team Name</th><th rowspan="2">Disease Risk Score AUC</th><th rowspan="2">Multi-Disease Average Score AUC + mAP</th><th rowspan="2">Final Score</th><th rowspan="2">Rank</th><th colspan="2">SC1</th><th colspan="2">SC2</th></tr><tr><th>F1 Score</th><th>Specificity</th><th>F1 Score</th><th>Specificity</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9875</td><td>0.7821</td><td>0.8848</td><td>1</td><td>0.9625</td><td>0.9668</td><td>0.5546</td><td>0.9957</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9861</td><td>0.7782</td><td>0.8821</td><td>2</td><td>0.9632</td><td>0.7836</td><td>0.5411</td><td>0.9876</td></tr><tr><td>BNUAA</td><td>0.9870</td><td>0.7578</td><td>0.8724</td><td>3</td><td>0.9593</td><td>0.9627</td><td>0.6516</td><td>0.9956</td></tr><tr><td>Nekaravuru</td><td>0.9782</td><td>0.7485</td><td>0.8634</td><td>4</td><td>0.9478</td><td>0.9403</td><td>0.5013</td><td>0.9871</td></tr><tr><td>WWW</td><td>0.9765</td><td>0.7380</td><td>0.8572</td><td>5</td><td>0.9170</td><td>0.9776</td><td>0.5089</td><td>0.9894</td></tr><tr><td>IGSTfencing</td><td>0.9797</td><td>0.7102</td><td>0.8449</td><td>6</td><td>0.9512</td><td>0.9254</td><td>0.5529</td><td>0.9923</td></tr><tr><td>MISIT</td><td>0.9706</td><td>0.6893</td><td>0.8300</td><td>7</td><td>0.9475</td><td>0.7899</td><td>0.6668</td><td>0.9891</td></tr><tr><td>chizu & arai & okada</td><td>0.9885</td><td>0.7796</td><td>0.8840</td><td>*</td><td>0.9506</td><td>0.9701</td><td>0.6445</td><td>0.9968</td></tr></tbody></table>

the challenge, these metrics were not utilized as evaluation parameters and, as a result, were not considered in the ranking of participating teams. The F1 score balances precision and recall, providing a comprehensive measure of a model's ability to correctly identify positive instances while minimizing false positives. Specificity, on the other hand, gauges a model's proficiency in correctly identifying true negatives, offering insights into its ability to avoid false alarms. Together, these metrics contribute to a more nuanced understanding of a model's effectiveness.

## 7. Results

This section reports and discusses the results of the challenge. Performance of all competing solutions on both sub-challenge including their leaderboard rank. The final rank on the leaderboard was determined by using performance in both sub-challenges as follows:

$$ \text{Final Score} = 0.5*(\text{Sub-challenge-1 Score}) + 0.5*(\text{Sub-challenge-2 Score}) $$

This section presents the performance of the proposed model by all participating teams for disease screening and classification tasks. Online result submission and automated evaluation system was built and presented on our website for the participants to submit and evaluate their results. Evaluated results are made publicly available on the leaderboard of our website. All the results submitted by competing teams were analyzed using the evaluation measures given in section 6. This measure generated the AUC score for sub-challenge-1 and AUC and mAP score for sub-challenge-2. We got total 74 submissions by individuals/teams for the challenge. Amongst them, the top performing 8 teams were invited to participate in the final online challenge.

Table 6 summarizes the best submission performance of the top 8 teams on the validation set. The column “Disease Risk Score” shows the performance of the individual solution for disease screening task (sub-challenge 1) and the column “Multi-Disease Average Score” gives the performance for the disease classification task (sub-challenge 2). The column “Final Score” is an aggregate performance of the individual solution on sub-challenge 1 and sub-challenge 2. The final placement for each subtask is determined using the “Final Score” from the leaderboard. Furthermore, we conducted an analysis for both tasks (SC1: sub-challenge-1 and SC2: sub-challenge-2), evaluating F1 score and Specificity. The performance of top 8 teams is reported in the same table. The “Final Score” of all the participating individual/ teams in decreasing order of their performance in sub-challenge-1 and sub-challenge-2 on validation set is shown in Fig.2 and 3 respectively. Fig.4 shows the distribution of “Final Score” in sub-challenge-2 across the top performing eight teams for all 28 disease categories on test set. The disease categories are ranked by decreasing order of their frequency of appearance in the RFMiD dataset.

Table 7 summarizes the individual performance on the test set of the top 8 teams selected based on the performance of their proposed solution on the validation set. Amongst the 8 teams on the day of the challenge, 7 teams submitted their results through the automated online evaluation system. The team chizu & arai & okada could not participate on the day of challenge but later submitted their results and thus they are not ranked. The top-performing solution on the day of challenge was proposed by team KAMATALAB followed by team Schulich Applied Computing inMed and team BNUAA with a final score of 0.8848, 0.8821, and 0.8724, respectively.

To provide a more comprehensive analysis of the top-
<figure><img src="image_7.png" /><figcaption>Fig. 2. The performance (Final Score) of all the participating individual/ teams in sub-challenge-1 on validation set.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 3. The performance (Final Score) of all the participating individual/ teams in sub-challenge-2 on validation set.</figcaption></figure>

performing results, ranking stability is employed, and boot-
strapping serves as a valuable technique for assessing the ro-
bustness of these rankings. Bootstrapping for ranking stability
is a statistical resampling technique used to assess the reliability
and variability of rankings. It involves repeatedly resampling
the observed data with replacement to create multiple simulated
datasets. Here we estimated 1,000 bootstrap samples. The 95%
confidence interval is determined by calculating the 2.5th and
97.5th percentiles of the distribution of the bootstrapped statis-
tic. This interval represents the range within which the true
population parameter is estimated to fall with 95% confidence
based on the variability observed in the resampled data. The

rankings are then recalculated and by comparing the original
ranking with those obtained from resampled datasets, one can
estimate the stability and uncertainty associated with the rank-
ings.

Tables 8 and 9 present bootstrap confidence intervals (Average, Lower, and Upper) for the best submission performances of the top 8 teams on the validation and test sets, respectively. The average represents the mean of bootstrapped values calculated over 1000 samples. In Table 8, the bootstrapped ranking closely aligns with the original ranking, with only slight alterations observed for ranks 4 (BNUAA) and 5 (Schulich Applied Computing inMed). The final bootstrapped ranking is deter-
<table><caption>Table 8. Bootstrap confidence intervals for best submission performance of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi-Disease Average Score) on validation set. AUC: area under the receiver operator curve, mAP: mean average precision, CI: confidence interval.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="3">Disease Risk Score<br>AUC<br>(95% CI) (Bootstrap)</th><th colspan="3">Multi-Disease<br>Average Score<br>AUC + mAP<br>(95% CI) (Bootstrap)</th><th colspan="3">Final Score<br>(Bootstrap)</th><th rowspan="2">Rank</th></tr><tr><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9724</td><td>0.9591</td><td>0.9827</td><td>0.8043</td><td>0.7821</td><td>0.8223</td><td>0.8884</td><td>0.8675</td><td>0.9151</td><td>1</td></tr><tr><td>IGSTfencing</td><td>0.9599</td><td>0.9456</td><td>0.9728</td><td>0.7981</td><td>0.77804</td><td>0.8171</td><td>0.8790</td><td>0.8673</td><td>0.8900</td><td>2</td></tr><tr><td>chizu & arai & okada</td><td>0.9639</td><td>0.9435</td><td>0.9757</td><td>0.7900</td><td>0.7725</td><td>0.8027</td><td>0.8770</td><td>0.8569</td><td>0.8947</td><td>3</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9611</td><td>0.9476</td><td>0.9740</td><td>0.7914</td><td>0.7718</td><td>0.8108</td><td>0.8763</td><td>0.8637</td><td>0.8878</td><td>5</td></tr><tr><td>BNUAA</td><td>0.9613</td><td>0.9481</td><td>0.9730</td><td>0.79149</td><td>0.7696</td><td>0.8111</td><td>0.8764</td><td>0.8646</td><td>0.8894</td><td>4</td></tr><tr><td>Nekaravuru</td><td>0.9627</td><td>0.9489</td><td>0.9746</td><td>0.76418</td><td>0.7424</td><td>0.7861</td><td>0.8634</td><td>0.8504</td><td>0.8761</td><td>6</td></tr><tr><td>WWW</td><td>0.9572</td><td>0.9425</td><td>0.9696</td><td>0.7627</td><td>0.7409</td><td>0.7841</td><td>0.8600</td><td>0.8469</td><td>0.8728</td><td>7</td></tr><tr><td>MISIT</td><td>0.9489</td><td>0.9316</td><td>0.9640</td><td>0.7085</td><td>0.6822</td><td>0.7354</td><td>0.8287</td><td>0.8120</td><td>0.8451</td><td>8</td></tr></tbody></table>

<table><caption>Table 9. Bootstrap confidence intervals of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi-Disease Average Score) on test set. AUC: area under the receiver operator curve, mAP: mean average precision, CI: confidence interval, p-value: performance of Wilcoxon signed-rank tests.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="3">Disease Risk Score<br>AUC<br>(95% CI) (Bootstrap)</th><th colspan="3">Multi-Disease<br>Average Score<br>AUC + mAP<br>(95% CI) (Bootstrap)</th><th colspan="3">Final Score<br>(Bootstrap)</th><th rowspan="2">Rank</th><th rowspan="2">p-value</th></tr><tr><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9880</td><td>0.9789</td><td>0.9928</td><td>0.7888</td><td>0.7676</td><td>0.8047</td><td>0.8885</td><td>0.8688</td><td>0.8947</td><td>1</td><td>0.005</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9861</td><td>0.9783</td><td>0.9923</td><td>0.7843</td><td>0.7565</td><td>0.8120</td><td>0.8853</td><td>0.8711</td><td>0.8996</td><td>2</td><td>0.014</td></tr><tr><td>BNUAA</td><td>0.9870</td><td>0.9794</td><td>0.9930</td><td>0.7651</td><td>0.7376</td><td>0.7952</td><td>0.8760</td><td>0.8620</td><td>0.8922</td><td>3</td><td>0.028</td></tr><tr><td>Nekaravuru</td><td>0.9784</td><td>0.9681</td><td>0.9876</td><td>0.7559</td><td>0.7250</td><td>0.7865</td><td>0.8671</td><td>0.8509</td><td>0.8825</td><td>4</td><td>0.056</td></tr><tr><td>WWW</td><td>0.9765</td><td>0.9667</td><td>0.9851</td><td>0.7454</td><td>0.7128</td><td>0.7743</td><td>0.8610</td><td>0.8436</td><td>0.8759</td><td>5</td><td>5.8e-13</td></tr><tr><td>IGSTfencing</td><td>0.9796</td><td>0.9703</td><td>0.9875</td><td>0.7186</td><td>0.6820</td><td>0.7531</td><td>0.8491</td><td>0.8305</td><td>0.8678</td><td>6</td><td>1.2e-07</td></tr><tr><td>MISIT</td><td>0.9236</td><td>0.8965</td><td>0.9474</td><td>0.6635</td><td>0.6309</td><td>0.6968</td><td>0.7936</td><td>0.7727</td><td>0.8142</td><td>7</td><td>-</td></tr><tr><td>chizu & arai & okada</td><td>0.9886</td><td>0.9822</td><td>0.9936</td><td>0.7880</td><td>0.7601</td><td>0.8150</td><td>0.8883</td><td>0.8746</td><td>0.9022</td><td>*</td><td>-</td></tr></tbody></table>

mined based on the overall average value. In Table 9, the bootstrapped ranking precisely mirrors the original ranking.

Kendall's tau (Kendall, 1938) correlation coefficient was employed to measure changes between the original and bootstrap rankings, ranging from 1 to -1. A coefficient of 1 signifies an unchanged ranking, while -1 indicates a reversed ranking (Wagner et al., 2023). The rankings appear stable, with the first and second algorithms closely together. This observation is reinforced by the mean Kendall's tau value across all bootstrap samples of 0.85 (median: 0.91, interquartile range (IQR): (0.81, 0.91)).

Additionally, pairwise Wilcoxon signed rank tests were conducted for multiple comparisons across algorithm pairs. The tests were used to check for algorithms that were significantly superior in terms of performance compared to their competitors (Wagner et al., 2023). The results of this analysis are independent from the bootstrap analysis. Multi-disease detection probabilities computed by two teams on test set data are compared using a Wilcoxon signed-rank test. The resulting p-values obtained for consecutive teams in the team ranking are reported in Table 9. The p-value < 0.05 suggest statistically significant evidence of median differences in performance data between the compared algorithms.

Finally, we tried both Averaging Ensemble and Weighted Averaging Ensemble for the top-performing methods (Table 10), but neither yielded satisfactory results compared to the individual methods. However, it's worth noting that model ensembling, which involves combining the outputs of multiple models, often offers advantages over simple probability averaging methods.

# 8. Discussion and Conclusion

## 8.1. Summary of the Challenge

In this paper, we have presented the details of the RIADD challenge, including detailed information about the dataset, the challenge organization, all the top-performing competing solutions, used evaluation metrics, and final individual and averaged results for both sub-tasks, i.e., disease screening and multi-disease classification.

The top performing methods for multi-disease detection demonstrated that the problem of multi-disease detection, limited data, and class imbalance can be handled using a substantial number of common tips: (1) appropriate data preprocessing, (2) data augmentation to handle the problem of data scarcity and class imbalance problem, (3) transfer learning using the pre-trained model, (4) modification of loss function, (5) model ensembling. Additionally, to improve model performance, many teams used the EfficientNet framework as the backbone model for the network, it learns gradient features by uniformly scaling all dimensions of depth/width/resolution using a compound coefficient. For both the sub-challenges, the best performing proposed network by team “KAMATALAB” along with these common tips, adopted post-processing steps to avoid over-fitting. As this is the first publicly available multi-disease dataset, most of the teams did not use an external dataset. The only team that used external unlabelled dataset is “IGSTfencing”, it used ODIR-2019 (ODIR, 2019) dataset and got 2nd and 6th position in sub-challenge 1 and sub-challenge 2, respectively.

Teams are observed (from tables 6 and 7) to perform poorly in the multi-disease classification task as compared to the dis-
<table><caption>Table 10. The performance of Averaging Ensemble and Weighted Averaging Ensemble for the top-performing methods on test set. AUC: area under the receiver operator curve, mAP: mean average precision, M1: KAMATA-LAB, M2: Schulich Applied Computing inMed, M3: BNUAA.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="3">Averaging Ensemble</th><th colspan="3">Weighted Averaging Ensemble</th></tr><tr><th>Disease Risk Score AUC</th><th>Multi-Disease Average Score AUC + mAP</th><th>Final Score</th><th>Disease Risk Score AUC</th><th>Multi-Disease Average Score AUC + mAP</th><th>Final Score</th></tr></thead><tbody><tr><td>M1 + M2</td><td>0.9877</td><td>0.7819</td><td>0.8848</td><td>0.9880</td><td>0.7830</td><td>0.8855</td></tr><tr><td>M1 + M2 + M3</td><td>0.9888</td><td>0.7812</td><td>0.8850</td><td>0.9885</td><td>0.7825</td><td>0.8855</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Fig. 4. Box plot showing the distribution of "Final Score" in sub-challenge-2 across the top performing eight teams for all 28 disease categories on test set. The disease categories are ranked by decreasing order of their frequency of appearance in the RFMiD dataset.</figcaption></figure>

ease screening task that reduced the overall final score. The major reason seems to be the difficulty in rare pathology classification which in turn decreases the overall performance of the multi-disease classification task. The same can also be seen from the box plot (Fig.4), showing the distribution of "Final Score" in sub-challenge-2 across the top performing eight teams for all 28 disease categories on test set. We observe that detection performance generally increases with the size of disease categories. Other factors probably come into play, such as the subtlety of diseases: optic disc pallor (ODP) is frequent but subtle, retinitis pigmentosa (RP) is rare but generally obvious. The 'Other' category, which is both small and varied, is particularly challenging. We note that the 8 teams have similar scores for frequent categories: the performance difference between teams largely comes from rare categories, where the variability between teams is high.

## 8.2. Limitations of the Study

This challenge provided data acquired in routine clinical practice using three different digital fundus cameras, using a consistent acquisition protocol for all images. The images captured with different retinal cameras and with multiple diseases

allowed a better estimation of the generalizability of the proposed architectures. The data was acquired either by ophthalmologists or an expert operator at two different centers. The images without an adjudicated consensus of expert annotator and non-gradable images are not included in the dataset. The period from data availability to the deadline for results submission, about three months, was considerable time for methodology development and submission for all tasks. Further, automated evaluation system have facilitated the submission and evaluation procedure by providing a continuous (real-time feedback) online assessment of participating solutions. This also enabled a maximum number of submissions during the challenge period. Even after these efforts to give the best possible data, adequate time for submission, and an automated evaluation system, the data annotation process is still intrinsically subjective, and the annotators' judgment is a restricting factor for the architecture performance. Further, data was acquired at one location only, for the generalizability of the network, it would be great if the data is from different ethnicity and race.
## 8.3. Future Work

Still, there is room for improvement, especially in multi-disease classification task. Though the competition is now completed, the dataset has been made publicly available for research purposes to attract newcomers to the problem and to encourage the development of novel solutions to meet current and future clinical standards. Additionally, it would be beneficial for the network's generalizability to obtain data from various ethnicities and races, considering that the current data was acquired at a single location.

## 8.4. Conclusion

This challenge not only identifies and addresses critical gaps in existing practices but also promotes collaboration, tackles real-world challenges, and encourages innovative solutions for multi-disease detection in retinal screening. Despite the complexity of the tasks, it received a very positive response and the top performing solutions were proposed by participating teams. Given the significant number of participating teams and the results obtained, we believe this challenge was a success. To the organizational end, efforts have been made in creating an admissible, challenging, and fair competition, capable of facilitating collective knowledge in the research community.

## Declaration of Competing Interest

The authors have no conflicts of interest to declare.

## Acknowledgments

This work is sponsored by the Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded (M.S.), INDIA, French National Research Agency under the LabCom program (ANR-19-LCV2-0005 - ADMIRE project), and OphtAI. The authors would like to thank Dhanashree Thulkar (Veermata Jijabai Technological Institute, Mumbai, India) for their help in various aspects of organizing the ISBI-2021 RIADD Challenge.

# Appendix A. Details of Competing Solutions

## Appendix A.1. KAMATALAB (Luo Zhengbo et al.)

Luo et al. proposed a diverse blending system based on EfficientNet (Tan and Le, 2019) for multi-disease detection as shown in Fig. A.5. It consists of three paths with various image sizes, versions of models, and training strategies (table A.11). The system did not use any extra data, and only a simple pre-processing, which cuts out the black edges, was applied. SGD was used with a Nesterov momentum of 0.9 as an optimizer, the learning rate was set to 0.001 initially and increased to 0.1 shortly, then divided by ten every ten epochs with an early stop (the best epoch usually between 15-20). The direct training strategy refers that: dealing with both sub-challenges together as a 29-classes task (1 class for sub-challenge one and 28 classes for sub-challenge two). The multi-stage strategy refers that: dividing sub-challenge two into three sub-tasks with a similar number of training samples, which are 9 classes

<figure><img src="image_10.png" /><figcaption>Fig. A.5. The overall structure of the proposed system for multi-disease detection.</figcaption></figure>

<table><caption>Table A.11. The details of three paths with various image sizes, version of models, and training strategies.</caption><thead><tr><td></td><td>Version of Model</td><td>Input size (in pixels)</td><td>Training strategy</td></tr></thead><tbody><tr><td>Path one</td><td>Efficient-Net B5</td><td>768 × 768 × 3</td><td>Multi-stage</td></tr><tr><td>Path two</td><td>Efficient-Net B5</td><td>960 × 960 × 3</td><td>Direct training</td></tr><tr><td>Path three</td><td>Efficient-Net B6</td><td>768 × 768 × 3</td><td>Direct training</td></tr></tbody></table>

with over 65 samples, 8 classes with samples number ranging from 20 to 65, and 11 classes under 20 samples. All models are trained through 5-fold cross-validation. The authors adopt both horizontal and vertical flip with $p = 0.5$, median-blur with blur limit 7, additive Gaussian noise which scales in a range of [0, 0.15 * 255], regular hue saturation and brightness augmentations, five holes cutout with max size [20, 20] and ImageNet normalization. However, resizing and imageNet normalization procedure took place for the validation. Test time augmentation (TTA) was applied for inference. Five times TTA for path one and three times TTA for path two and three with setting $p = 0.5$ for horizontal and vertical flip, hue, saturation, random brightness, and ImageNet normalization. The authors applied post-processing mainly on two aspects: minor classes and correlation between two sub-challenges. The post-processing applied are, average blending for major classes only, providing meta-Information, replacement with min-max values, and rank voting. Moreover, the risk score of sub-challenge one depends on whether the image has any disease in sub-challenge two. Decrease predictions with a low-risk score in sub-challenge two gave a reasonable raise in performance. The best off-site result was based on an average blending of all three paths; however, the on-site result was based on path two and three only due to the time limitation. The source code is available at https://github.com/Hanson0910/Pytorch-RIADD.

## Appendix A.2. Schulich Applied ComputingInMed (Edward Ho et al.)

Ho et al. used an ensemble of 5 CNN architectures to predict both disease risk (presence or absence of any pathologies) and disease classification simultaneously in a multilabel, multiclass problem formulation with 29 classes. All models were trained with preprocessed images that were square cropped around the center of the image (TOPCON) or cropped by finding the rectangular area containing all pixels above a constant threshold value (Kowa). The 5 architectures used were SE-ResNeXt (Hu
et al., 2018b), InceptionV3 (Szegedy et al., 2016), EfficientNet-B4 and B5 (Tan and Le, 2019), and DenseNet (Huang et al., 2017). For the SE-ResNeXt model, a dropout layer was added before the last linear layer. All other architectures were un-modified. The hyperparameters and image input sizes for the 5 different architectures are given in table A.12.

<table><caption>Table A.12. Hyperparameters and image input sizes for the 5 different architectures.</caption><thead><tr><th>Architecture</th><th>Image Size (in pixels)</th><th>Batch Size</th><th>Learning Rate</th><th>Training Epoch</th></tr></thead><tbody><tr><td>SE-ResNeXt</td><td>512</td><td>8</td><td>1 × 10<sup>-4</sup></td><td>20</td></tr><tr><td>DenseNet-121</td><td>299</td><td>16</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr><tr><td>InceptionV3</td><td>299</td><td>32</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr><tr><td>EfficientNet-B4</td><td>380</td><td>12</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr><tr><td>EfficientNet-B5</td><td>456</td><td>8</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr></tbody></table>

Networks were all trained by optimizing an asymmetric loss, as described by Ben-Baruch et al. (Ben-Baruch et al., 2020), to counter the significant class imbalances found in this problem. This is a modified BCE loss such that the loss is defined as:

$$L = -yL_{+} - (1 - y)L_{-} \quad (A.1)$$

where $L_+$ and $L_-$ represent the positive and negative loss parts and are defined as:

$$ \begin{aligned} L_{+} &= (1 - p)^{y_{+}} \log(p) \\ L_{-} &= (p_{m})^{y_{-}} \log(1 - p_{m}) \end{aligned} \quad (A.2) $$

where $y_+$ and $y_-$ are the positive and negative focusing parameters, respectively; and $p_m$ is the shifted probability defined by:

$$p_m = \max(p - m, 0) \quad (A.3)$$

With the shifted probability in the negative loss part, this allows easy negative examples (such that the prediction probability is less than the probability margin $m$) to not factor into the loss. By setting $y_- > y_+$, the contribution of positive samples to the loss can be increased. The focusing parameters and probability margins are tunable hyperparameters.

All networks were initialized by pre-training on ImageNet and trained using the Adam (Kingma and Ba, 2014). Random horizontal flip and random rotation (between -30 and 30 degrees) augmentations were used for all networks except InceptionV3, which rotated between -90 and 90 degrees and incorporated a random brightness transformation.

For each of the 5 CNN architectures, models, were trained with a 5-fold cross-validation on iteratively stratified folds (Sechidis et al., 2011), (Szymański and Kajdanowicz, 2017) to preserve the distribution of class labels. This results in 5 independent models, and the predicted scores are averaged to serve as the output for a given architecture. Finally, the outputs from all five architectures are averaged to form the final ensemble.

## Appendix A.3. BNUAA (Xinliang Wang et al.)

Wang et al. proposed a novel Disease Head to solve the disease screening and multi-disease detection problems. The authors designed a simple method to binarize the original image

and used the binary image as a mask to delete the invalid background in the original fundus images. In addition, the authors used heavy augmentation resize, transpose, horizontal flip, vertical flip, random brightness contrast, hue saturation value, shift scale rotate, coarse dropout, cut out, and mixup to avoid overfitting. The Ranger optimizer and CosineAnnealingLR learning rate scheduler were employed to achieve faster convergence speed and better results. There are 2 categories in the sub-challenge 1: have fundus diseases or not, and 28 categories in sub-challenge 2: different fundus diseases. The authors found that these two tasks are related because any disease found in task 2 would show positively in task 1. Thus, a Disease Head was proposed which utilizes this prior knowledge. Fig. A.6

<figure><img src="image_11.png" /><figcaption>Fig. A.6. The BNUAA designed module.</figcaption></figure>

shows the designed module, it is used as the classification head in CNN. The loss function for sub-challenge 1 is the cross-entropy (CE) loss. For sub-challenge 2 the authors used the BCE loss as the loss function. In addition, when using the proposed Disease Head, the loss function of sub-challenge 1 is modified to $\alpha L_{CE} + \beta L_{BCE}$. In the experiment, the ResNet (He et al., 2016) and ResNext (Xie et al., 2017) were used as the backbone and resized the image size to 512 × 512 pixels. The model was implemented in PyTorch and Python 3.6. The learning rate started from 0.001 and the networks were trained on one Tesla V100 with 40 epochs.

## Appendix A.4. Nekaravuru (Young-tack Oh et al.)

Oh et al. developed a two-branch network based on Efficient-Net (Tan and Le, 2019). Each branch was named LP for classifying large positive classes and SP for classifying small positive classes. The large positive class and small positive class were divided based on whether the number of positive samples exceeded 40 in the total data. Initially, for the given training

<figure><img src="image_12.png" /><figcaption>Fig. A.7. Pre-processing for three digital fundus camera images.</figcaption></figure>
images, data preprocessing was performed (Fig. A.7) and flip, rotation, shifting, blur, elastic transformations, grid distortion, and cutout data augmentation. The authors also set M_with that includes ‘Disease Risk’ class and MWithout that excludes ‘Disease Risk’ class based on these divided classes. The final result is an ensemble of M_with and MWithout (Fig. A.8).

<figure><img src="image_13.png" /><figcaption>Fig. A.8. LP is the classifier for large positive class and SP is the classifier for small positive class. M_with is composed of 13 class (a), and MWithout is composed of 12 class (a*).</figcaption></figure>

Since negative labels dominate positive labels in most classes, the loss function should be adjusted to be more sensitive to detecting positive labels (Murphy, 2012). Thus, the authors used positive weighted binary cross entropy as a loss function given by:

$$L_c = \frac{1}{N} \sum_{n=0}^{N} [-\alpha_c \cdot y_{c,n} \cdot \log\sigma(x_{c,n}) - (1 - y_{c,n}) \cdot \log(1 - \sigma(x_{c,n}))] \quad (A.4)$$

Where c is the class number, N is the batch size, yc,n and xc,n represent the true label and predicted label for the c class for the nth image. The positive weights αc for class c is defined as follows:

$$\alpha_c = \frac{L_c^{\text{neg}}}{L_c^{\text{pos}}} \quad (A.5)$$

Where L_c^neg and L_c^pos represent the number of negative labels and the number of positive labels for class c. However, if αc is specified as above, the learning did not perform well because too much weight could be given to the little amount of data. So, authors set the threshold T_α and αc is defined as follows:

$$\alpha_c = \begin{cases} 7 & \text{if } T_\alpha < \alpha_c \\ 3 & \text{if } 3 < \alpha_c < T_\alpha \\ 1 & \text{otherwise} \end{cases} \quad (A.6)$$

In the large positive class, T_α was set to 10, and in the small positive class, T_α was set to 50. The final loss Ltotal is as follows:

$$L_{\text{total}} = \beta \cdot L_L + (1 - \beta)L_S \quad (A.7)$$

Where L_L and L_S represent the loss of LP classifier and the loss of SP classifier, respectively (Fig. A.8). β is a hyperparameter to indicate the loss ratio for each classifier. Experimentally, a large difference in the ratio gave worse results, and finally, β = 0.55 was set.

The method was implemented based on Pytorch and trained on four NVIDIA GeForce GTX 1080 GPUs. The Adam optimization algorithm and the initial learning rate is 3 × 10⁻⁴ was used. A Plateau scheduler was used that multiplies the learning rate by 0.7 whenever the learning area under the receiver operator curve (AUC) stagnates. EfficientNet pre-trained with ImageNet as a feature extractor was used.

Matsunaga et al. and Jin et al. have empirically found that combining predictions for multiple transformed versions of the test image helps improve performance (Matsunaga et al., 2017; Radosavovic et al., 2018; Wang et al., 2019). For this reason, the augmentation techniques that are applied during learning to the test image were able to get better results. In addition, the authors trained and ensembled a model with three different versions (EfficientNet-b0, b1, b2) of feature extractors to obtain a more generalized model.

## Appendix A.5. WWW (Hung Yeh et al.)

Yeh et al. first removed the redundant black edge from the fundus images. Then, rescaled images from the dataset to a size of 512 × 512 pixels as the input image. Then, CLAHE) and green channel of the image were used as pre-processing methods to improve the brightness and contrast. In addition, two functions of AutoAugment (Cubuk et al., 2019) to enhance the RFMiD dataset, called flip and rotation were used.

The authors used ImageNet pre-trained model and adopted EfficientNet (Tan and Le, 2019) and TResNet (Ridnik et al., 2021) to detect the ocular disease and classify the categorization of ocular disease. Moreover, different versions of EfficientNet were used for training, and the ensemble process was implemented to promote high performance. In the convolution layer of the deep learning network, the ReLU activation function was replaced with Mish (Misra, 2019) and used a ranger optimizer rather than SGD. Authors used an asymmetric loss function, as described by Ben-Baruch et al. (Ben-Baruch et al., 2020) to tackle the imbalanced data.

There are two sub-challenge in this challenge, namely, disease screening and disease classification. The authors used two kinds of image processing algorithms to provide different datasets and six versions of EfficientNet for experiments. For these two sub-challenge, the authors found that Efficient-Net B7 performed the best, which gets the best generalization and improved the prediction robustness of the deep learning network. In the disease classification task, the proposed algorithm achieved a better performance than the disease screening task. However, there are several categorizations in the disease screening task, which made it more difficult. After observing, the authors found that every single model had a specific ability in different tasks. Therefore, an ensemble algorithm was adopted
to merge the advantage of multiple diverse models to predict an outcome.

## Appendix A.6. IGSTfencing (Haoyu Wang et al.)

Wang et al. proposed a semi-supervised siamese graph convolutional network (SS-GCN) for both sub-challenges 1 and 2. SENet154 (Hu et al., 2018c) pre-trained on ImageNet dataset was used as the backbones for the proposed SS-GCN. The model adopted an MLCR-style (Niu et al., 2020) semi-supervised framework with two branches to learn useful information from both labeled and unlabeled data. Motivated by SSGRL (Chen et al., 2019), a semantic encoder was designed to extract feature vectors for each category based on both image feature maps and semantic information. Given a retinal image, backbone CNN was deployed together with semantic encoder to get semantic feature vectors of each branch. Then, feature vectors from two branches are fed into a GCN to model label relation. Finally, two outputs are fused to predict the final label distribution. The pipeline of the proposed SS-GCN framework is shown in Fig. A.9.

<figure><img src="image_14.png" /><figcaption>Fig. A.9. The pipeline of proposed semi-supervised siamese graph convolutional network (SS-GCN) framework. Both labeled and unlabeled images are encoded with the semantic encoder to obtain semantic features. Then, the multi-view GCN incorporates the features to model relations between each category to construct the final classifier. Finally, the feature vector after GAP is fed into the classifier to get the final results. $L_{mv}$, $L_{ce}$, and $L_{cr}$ are terms of the loss function to regularize different parts of SS-GCN.</figcaption></figure>

The BCE loss function was used for the backbone CNN and semantic encoder of each branch. In the semi-supervised framework, multi-view and co-regularization loss functions like MLCR for semi-supervised learning were adopted. The final loss function of the proposed SS-GCN framework can be formulated as:

$$L = \frac{1}{2} \sum_{i=0}^{1} L_{ce}^{i} + \lambda_{mv} L_{mv} + \lambda_{cr} L_{cr}, \quad (A.8)$$

The model was trained for 25 epochs using a SGD optimizer with an initial learning rate $\alpha_0^{(1)} = 0.07$ for multi-view GCN and $\alpha_1^{(1)} = 1.5 \times \alpha_0^{(1)}$ for semantic encoders to update the weights of the network. The learning rate was decayed as follows:

$$\alpha_i^{(n)} = 0.1^{\lfloor n/16 \rfloor} \times \alpha_i^{(1)}. \quad (A.9)$$

The learning rate is reduced by a factor of 0.1 at 16 epochs. Balancing factors $\lambda_{mv}$ and $\lambda_{cr}$ of loss function are set to 9.5. The authors utilize unlabeled data from ODIR-2019 (ODIR,

2019) which is publicly provided by Peking University. The batch size of each iteration is set to 16 (8 labeled images and 8 unlabeled images). All the experiments are conducted on an NVIDIA TESLA V100 GPU.

Before feeding the images into the network, the ROI was cropped from raw retinal images and resized into size (448×448 pixels). Then, three types of data augmentation were deployed to prevent the over-fitting issue. Firstly, a random horizontal flip was applied with a probability of 0.5 because it is ambiguous to distinguish whether the retinal image is from the left eye or the right eye. Secondly, random rotation was applied with a range of degrees in [-30, 30] because the shape of ROI is central symmetry. Thirdly, the network was trained by randomly cropping the image with a scale [0.5, 1.2] of (448 × 448 pixels) to enhance its generalization performance. Finally, the input image was ImageNet normalized. For inference of the evaluation set, a simple TTA strategy was used. First the images were resized to (512×512 pixels), then the center region with size (448×448 pixels) was cropped and then were ImageNet normalized. Then, flip the image horizontally, feed images of two directions into the network, and compute the mean value of outputs as the final result.

## Appendix A.7. MISIT (Dominik Müller et al.)

Müller et al. developed a multi-disease detection pipeline for retinal imaging which focuses on ensemble learning to combine the predictive capabilities of several heterogeneous deep CNN models. Their pipeline (Fig. A.10) consisted of three core pillars: The preprocessing methods, the neural network models, and the ensemble learning techniques.

<figure><img src="image_15.png" /><figcaption>Fig. A.10. Flowchart diagram of the implemented pipeline from Müller et al. for retinal multi-disease detection.</figcaption></figure>

For the augmentation, Müller et al. applied an up-sampling image augmentation approach (rotation, flipping, brightness, saturation, contrast, and hue) against the strong class imbalance bias, image padding to ensure aspect ratios and avoid distortion during the resizing, cropping to focus on the relevant retinal section, image resizing to 244 × 244, 299 × 299, or 380 × 380 pixels depending on the utilized neural network architecture and image intensity value normalization via Z-Score based on the average and standard deviation from ImageNet (Russakovsky et al., 2015). Next to the up-sampling image augmentation, images were real-time augmented during the training process to decrease the risk of overfitting and increase model generalizability.
As neural network models, the authors created a collection of deep learning models for different types of classification tasks based on different deep learning architectures. For disease risk detection, the DenseNet201, and EfficientNet B4 architecture were utilized, whereas the DenseNet201, EfficientNet B4, ResNet152, and InceptionV3 were used for disease-label annotation of abnormal images (Huang et al., 2017; Tan and Le, 2019; Szegedy et al., 2016; He et al., 2016). The model training process utilized a transfer learning strategy based on ImageNet, a dynamic learning rate starting from $10^{-4}$ to a maximum decrease of $10^{-7}$, an early stopping, as well as a model checkpoint approach to reduce the risk of overfitting and a class, weighted focal loss implementation (Russakovsky et al., 2015; Lin et al., 2017). Instead of the traditional epoch definition as a single run through the complete training data, the authors defined an epoch as 250 iterations (batches). The training was performed for 290 epochs with a batch size of 24 images.

In addition to multiple model architectures and individual tasks, the authors applied a stratified multi-label 5-fold cross-validation for model training. This bagging technique allowed a more efficient usage of training data which resulted in a collection of in total 30 deep learning models. Finally, the pipeline ensembles the predictions of all deep learning models through an ensemble learning stacking technique based on binary logistic regression models.

The pipeline was implemented with their in-house developed framework AUCMEDI using two NVIDIA Titan RTX GPUs. To ensure full reproducibility, the authors published their complete code on GitHub: https://github.com/frankkramer-lab/riadd.aucmedi.

## Appendix A.8. chizu & arai & okada (Hidehisa Arai et al.)

Arai et al. proposed a standard ensemble framework to combine predictions of several CNNs. To make an effective ensemble, the authors made models differ from each other by choosing models with diverging complexity and training them on different image resolutions. For each of the 29 targets, different blending weights were adjusted to maximize scores on local cross-validation. The images in the given data set were taken by three different cameras, each with different aspect ratios and retinal reflections. Therefore, the authors trimmed the background using a circle crop and then aligned the aspect ratio. Figure A.11 shows an example of fundus images taken by three different cameras and the pre-processed images applied to them. Then, the images were resized to 512 × 512 pixels.

Because of data scarcity especially, for rare classes, data augmentation plays an important role in this competition. The authors applied four different data augmentations to the images. To make models robust to the change of position, scale, and rotation, the authors used random resized crop and translate, scale, and rotate, and to make models invariant to the change of hue and brightness, the authors used random brightness contrast and hue saturation. Several models were chosen from EfficientNet (Tan and Le, 2019) families and NFNet (Brock et al., 2021) families and trained with several image resolution. Focal Loss (Lin et al., 2017) was employed to train models. Focal Loss dynamically weights the contribution of each sample

<figure><img src="image_16.png" /><figcaption>Fig. A.11. Example of fundus images and results of pre-processing on them.</figcaption></figure>

on the loss function based on prediction difficulty, thus resulting in better performance on the prediction to the classes with fewer samples. Adam wrapped was used in SAM (Foret et al., 2020) optimizer. SAM simultaneously minimizes loss value and loss sharpness, so the models became more robust. The initial learning rate was set to 0.001 and changed using a learning rate scheduling technique called Cosine Annealing. Cosine Annealing changes learning rate $\eta_t$ to follow the value of the following function:

$$ \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min}) \left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right) \quad (A.10) $$

The authors set $\eta_{min} = 0.0$, $\eta_{max} = 0.001$, $T_{max} = 10$ and trained the models for 55 epochs. To ensemble the predictions of each model, weighted blending was used. Blending weights are adjusted for each target. The random search was applied to optimize the weights by maximizing the local cross-validation score for each target. Implementation of this work was done in PyTorch v1.7. The training was performed using an NVIDIA GeForce RTX 3090, two NVIDIA Tesla V100 with 32GB vRAM, and one NVIDIA GeForce RTX 2080. The source code for training is available at https://github.com/koukyo1994/riadd-competition and https://github.com/Chizuchizu/riadd. The source code for inference is available at https://www.kaggle.com/hidehisaarai1213/riadd-inference.

## References

* Abràmoff, M.D., Lou, Y., Erginay, A., Clarida, W., Amelon, R., Folk, J.C., Niemeijer, M., 2016. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Investigative ophthalmology & visual science 57, 5200-5206.

* Ahn, J.M., Kim, S., Ahn, K.S., Cho, S.H., Lee, K.B., Kim, U.S., 2018. A deep learning model for the detection of both advanced and early glaucoma using fundus photography. PloS one 13, e0207982.
Ben-Baruch, E., Ridnik, T., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik-Manor, L., 2020. Asymmetric loss for multi-label classification. arXiv preprint arXiv:2009.14119.

Brock, A., De, S., Smith, S.L., Simonyan, K., 2021. High-performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171.

Chang, Y.S., Weng, S.F., Chang, C., Wang, J.J., Tseng, S.H., Wang, J.Y., Jan, R.L., 2016. Risk of retinal vein occlusion following end-stage renal disease. Medicine 95.

Chen, T., Xu, M., Hui, X., Wu, H., Lin, L., 2019. Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition. arXiv:1908.07325 [cs] arXiv:1908.07325.

Choi, J.Y., Yoo, T.K., Seo, J.G., Kwak, J., Um, T.T., Rim, T.H., 2017. Multi-categorical deep learning neural network to classify retinal images: A pilot study employing small database. PloS one 12, e0187336.

Christopher, M., Belghith, A., Bowd, C., Proudfoot, J.A., Goldbaum, M.H., Weinreb, R.N., Girkin, C.A., Liebmann, J.M., Zangwill, L.M., 2018. Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs. Scientific reports 8, 1-13.

Cuadros, J., Bresnick, G., 2009. Eyepacs: an adaptable telemedicine system for diabetic retinopathy screening. Journal of diabetes science and technology 3, 509-516.

Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V., 2019. Autoaugmentation: Learning augmentation strategies from data, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123.

Decencière, E., Zhang, X., Cazuguel, G., Lay, B., Cochener, B., Trone, C., Gain, P., Ordonez, R., Massin, P., Erginay, A., et al., 2014. Feedback on a publicly distributed image database: the messidor database. Image Analysis & Stereology 33, 231-234.

DeVries, T., Taylor, G.W., 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552.

Diaz-Pinto, A., Morales, S., Naranjo, V., Köhler, T., Mossi, J.M., Navea, A., 2019. Cnn's for automatic glaucoma assessment using fundus images: an extensive validation. Biomedical engineering online 18, 1-19.

Farnell, D.J., Hatfield, F., Knox, P., Reakes, M., Spencer, S., Parry, D., Harding, S.P., 2008. Enhancement of blood vessels in digital fundus photographs via the application of multiscale line operators. Journal of the Franklin institute 345, 748-765.

Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B., 2020. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412.

Fu, H., Li, F., Orlando, J., Bogunovic, H., Sun, X., Liao, J., Xu, Y., Zhang, S., Zhang, X., 2020. Adam: Automatic detection challenge on age-related macular degeneration. IEEE Dataport.

Fumero, F., Alayón, S., Sanchez, J.L., Sigut, J., Gonzalez-Hernandez, M., 2011. Rim-one: An open retinal image database for optic nerve evaluation, in: 2011 24th international symposium on computer-based medical systems (CBMS), IEEE. pp. 1-6.

Gargeya, R., Leng, T., 2017. Automated identification of diabetic retinopathy using deep learning. Ophthalmology 124, 962-969.

Grassmann, F., Mengelkamp, J., Brandl, C., Harsch, S., Zimmermann, M.E., Linkohr, B., Peters, A., Heid, I.M., Palm, C., Weber, B.H., 2018. A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology 125, 1410-1420.

Group, A.R.E.D.S.R., et al., 2001. A randomized, placebo-controlled, clinical trial of high-dose supplementation with vitamins c and e, beta carotene, and zinc for age-related macular degeneration and vision loss: Ares report no. 8. Archives of ophthalmology 119, 1417-1436.

Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al., 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316, 2402-2410.

He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.

Hoover, A., Kouznetsova, V., Goldbaum, M., 2000. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medical imaging 19, 203-210.

Hu, J., Chen, Y., Zhong, J., Ju, R., Yi, Z., 2018a. Automated analysis for

retinopathy of prematurity by deep neural networks. IEEE transactions on medical imaging 38, 269-279.

Hu, J., Shen, L., Sun, G., 2018b. Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141.

Hu, J., Shen, L., Sun, G., 2018c. Squeeze-and-excitation networks.

Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708.

Keel, S., Wu, J., Lee, P.Y., Scheetz, J., He, M., 2019. Visualizing deep learning models for the detection of referable diabetic retinopathy and glaucoma. JAMA ophthalmology 137, 288-292.

Kendall, M.G., 1938. A new measure of rank correlation. Biometrika 30, 81-93.

Kingma, D.P., Ba, J., 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

Li, Z., He, Y., Keel, S., Meng, W., Chang, R.T., He, M., 2018. Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology 125, 1199-1206.

Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P., 2017. Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, pp. 2980-2988.

Liu, R., Wang, X., Wu, Q., Dai, L., Fang, X., Yan, T., Son, J., Tang, S., Li, J., Gao, Z., et al., 2022. Deepdrid: Diabetic retinopathy—grading and image quality estimation challenge. Patterns 10, 100512.

MacGillivray, T., Trucco, E., Cameron, J., Dhillon, B., Houston, J., Van Beek, E., 2014. Retinal imaging as a source of biomarkers for diagnosis, characterization and prognosis of chronic illness or long-term conditions. The British journal of radiology 87, 20130832.

Mansour, R.F., 2018. Deep-learning-based automatic computer-aided diagnosis system for diabetic retinopathy. Biomedical engineering letters 8, 41-57.

Matsunaga, K., Hamada, A., Minagawa, A., Koga, H., 2017. Image classification of melanoma, nevus and seborrheic keratosis by deep neural network ensemble. arXiv preprint arXiv:1703.03108.

Misra, D., 2019. Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:1908.08681.

Mookiah, M.R.K., Chua, C.K., Min, L.C., Ng, E., Laude, A., 2013. Computer aided diagnosis of diabetic retinopathy using multi-resolution analysis and feature ranking frame work. Journal of Medical Imaging and Health Informatics 3, 598-606.

Murphy, K.P., 2012. Machine learning: a probabilistic perspective. MIT press.

Nielsen, K.B., Lautrup, M.L., Andersen, J.K., Savarimuthu, T.R., Grauslund, J., 2019. Deep learning-based algorithms in screening of diabetic retinopathy: A systematic review of diagnostic performance. Ophthalmology Retina 3, 294-304.

Niemeijer, M., Van Ginneken, B., Cree, M.J., Mizutani, A., Quellec, G., Sánchez, C.I., Zhang, B., Hornero, R., Lamard, M., Muramatsu, C., et al., 2009. Retinopathy online challenge: automatic detection of microaneurysms in digital color fundus photographs. IEEE transactions on medical imaging 29, 185-195.

Niu, X., Han, H., Shan, S., Chen, X., 2020. Multi-label Co-regularization for Semi-supervised Facial Action Unit Recognition. arXiv:1910.11012 [cs] arXiv:1910.11012.

OAM, B.A.H.P.D., Resnikoff, S., 2002. The role of optometry in vision 2020. Community Eye Health 15, 33-36.

ODIR, 2019. challenge. https://odir2019.grand-challenge.org/.

ODIR-2019, 2019. Peking university international competition on ocular disease intelligent recognition (odir-2019). IEEE Dataport.

Orlando, J.I., Fu, H., Breda, J.B., van Keer, K., Bathula, D.R., Diaz-Pinto, A., Fang, R., Heng, P.A., Kim, J., Lee, J., et al., 2020. Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Medical image analysis 59, 101570.

Pachade, S., Porwal, P., Kokare, M., Giancardo, L., Mériaudeau, F., 2021a. Neten: Nested efficientnet and adversarial learning for joint optic disc and cup segmentation. Medical Image Analysis 74, 102253.

Pachade, S., Porwal, P., Thulkar, D., Kokare, M., Deshmukh, G., Sahasrabud-dhe, V., Giancardo, L., Quellec, G., Mériaudeau, F., 2021b. Retinal fundus multi-disease image dataset (rfmid): a dataset for multi-disease detection research. Data 6, 14.

PALM-2019, 2019. Palm challenge. IEEE Dataport.

Patton, N., Aslam, T.M., MacGillivray, T., Deary, I.J., Dhillon, B., Eikelboom, R.H., Yogesan, K., Constable, I.J., 2006. Retinal image analysis: concepts,
applications and potential. Progress in retinal and eye research 25, 99–127.

Pead, E., Megaw, R., Cameron, J., Fleming, A., Dhillon, B., Trucco, E., MacGillivray, T., 2019. Automated detection of age-related macular degeneration in color fundus photography: a systematic review. survey of ophthalmology 64, 498–511.

Peng, Y., Keenan, T.D., Chen, Q., Agrón, E., Allot, A., Wong, W.T., Chew, E.Y., Lu, Z., 2020. Predicting risk of late age-related macular degeneration using deep learning. NPJ digital medicine 3, 1–10.

Phan, S., Satoh, S., Yoda, Y., Kashiwagi, K., Oshika, T., 2019. Evaluation of deep convolutional neural networks for glaucoma detection. Japanese journal of ophthalmology 63, 276–283.

Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabud-dhe, V., Meriaudeau, F., 2018a. Indian diabetic retinopathy image dataset (idrid). URL: https://dx.doi.org/10.21227/H25W98, doi:10.21227/H25W98.

Porwal, P., Pachade, S., Kokare, M., Deshmukh, G., Son, J., Bae, W., Liu, L., Wang, J., Liu, X., Gao, L., et al., 2020. Idrid: Diabetic retinopathy-segmentation and grading challenge. Medical image analysis 59, 101561.

Porwal, P., Pachade, S., Kokare, M., Giancardo, L., Mériaudeau, F., 2018b. Retinal image analysis for disease screening through local tetra patterns. Computers in biology and medicine 102, 200–210.

Quellec, G., Lamard, M., Conze, P.H., Massin, P., Cochener, B., 2020. Automatic detection of rare pathologies in fundus photographs using few-shot learning. Medical image analysis 61, 101660.

Quellec, G., Lamard, M., Lay, B., Guilcher, A.L., Erginay, A., Cochener, B., Massin, P., 2019. Instant automatic diagnosis of diabetic retinopathy. arXiv preprint arXiv:1906.11875.

Radosavovic, I., Dollár, P., Girshick, R., Gkioxari, G., He, K., 2018. Data distillation: Towards omni-supervised learning. in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4119–4128.

Raghavendra, U., Fujita, H., Bhandary, S.V., Gudigar, A., Tan, J.H., Acharya, U.R., 2018. Deep convolution neural network for accurate diagnosis of glaucoma using digital fundus images. Information Sciences 441, 41–49.

Raju, M., Pagidimarri, V., Barreto, R., Kadam, A., Kasivajjala, V., Aswath, A., 2017. Development of a deep learning algorithm for automatic diagnosis of diabetic retinopathy, in: MEDINFO 2017: Precision Healthcare through Informatics. IOS Press, pp. 559–563.

Ridnik, T., Lawen, H., Noy, A., Ben Baruch, E., Sharir, G., Friedman, I., 2021. Tresnet: High performance gpu-dedicated architecture, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1400–1409.

Roychowdhury, S., Koozekanani, D.D., Parhi, K.K., 2013. Dream: diabetic retinopathy analysis using machine learning. IEEE journal of biomedical and health informatics 18, 1717–1728.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al., 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211–252.

Sechidis, K., Tsoumakas, G., Vlahavas, I., 2011. On the stratification of multi-label data, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, pp. 145–158.

Shibata, N., Tanito, M., Mitsuhashi, K., Fujino, Y., Matsuura, M., Murata, H., Asaoka, R., 2018. Development of a deep residual learning algorithm to screen for glaucoma from fundus photography. Scientific reports 8, 1–9.

Shortliffe, E.H., Blois, M.S., 2006. The computer meets medicine and biology: emergence of a discipline, in: Biomedical Informatics. Springer, pp. 3–45.

Sivaswamy, J., Krishnadas, S., Chakravarty, A., Joshi, G., Tabish, A.S., et al., 2015. A comprehensive retinal image dataset for the assessment of glaucoma from the optic nerve head analysis. JSM Biomedical Imaging Data Papers 2, 1004.

Son, J., Shin, J.Y., Kim, H.D., Jung, K.H., Park, K.H., Park, S.J., 2020. Development and validation of deep learning models for screening multiple abnormal findings in retinal fundus images. Ophthalmology 127, 85–94.

Staal, J., Abrámoff, M.D., Niemeijer, M., Viergever, M.A., Van Ginneken, B., 2004. Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medical imaging 23, 501–509.

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016. Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826.

Szymański, P., Kajdanowicz, T., 2017. A network perspective on stratification of multi-label data, in: First International Workshop on Learning with Imbalanced Domains: Theory and Applications, PMLR. pp. 22–35.

Tan, J.H., Bhandary, S.V., Sivaprasad, S., Hagiwara, Y., Bagchi, A., Raghavendra, U., Rao, A.K., Raju, B., Shetty, N.S., Gertych, A., et al., 2018. Age-related macular degeneration detection using deep convolutional neural network. Future Generation Computer Systems 87, 127–135.

Tan, M., Le, Q., 2019. Efficientnet: Rethinking model scaling for convolutional neural networks, in: International Conference on Machine Learning, PMLR. pp. 6105–6114.

Ting, D.S.W., Cheung, C.Y.L., Lim, G., Tan, G.S.W., Quang, N.D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I.Y., Lee, S.Y., et al., 2017. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. Jama 318, 2211–2223.

de Vente, C., Vermeer, K.A., Jaccard, N., van Ginneken, B., Lemij, H.G., Sánchez, C.I., 2021. Rotterdam eyepacs airogs train set. URL: https://doi.org/10.5281/zenodo.5793241, doi:10.5281/zenodo.5793241. The previous version was split into two records. This new version contains all data and the second record is deprecated.

Wagner, M., Müller-Stich, B.P., Kisilenko, A., Tran, D., Heger, P., Mündermann, L., Lubotsky, D.M., Müller, B., Davitashvili, T., Capek, M., et al., 2023. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the heichole benchmark. Medical image analysis 86, 102770.

Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., 2019. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing 338, 34–45.

Wang, J., Ju, R., Chen, Y., Zhang, L., Hu, J., Wu, Y., Dong, W., Zhong, J., Yi, Z., 2018. Automated retinopathy of prematurity screening using deep neural networks. EBioMedicine 35, 361–368.

WHO, 2019. World report on vision. World Health Organization.

Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K., 2017. Aggregated residual transformations for deep neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492–1500.

Yang, W., Xu, H., Yu, X., Wang, Y., 2015. Association between retinal artery lesions and nonalcoholic fatty liver disease. Hepatology international 9, 278–282.

Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D., 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.
<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

Medical Image Analysis (2024)

Contents lists available at ScienceDirect

Medical Image Analysis

journal homepage: www.elsevier.com/locate/media

<figure><img src="image_19.png" /></figure>

RFMiD: Retinal Image Analysis for multi-Disease Detection Challenge

Samiksha Pachade<sup>a,1,*</sup>, Prasanna Porwal<sup>a,1</sup>, Manesh Kokare<sup>a,1</sup>, Girish Deshmukh<sup>b,1</sup>, Vivek Sahasrabuddhe<sup>c,1</sup>, Zhengbo Luo<sup>d</sup>, Feng Han<sup>e</sup>, Zitang Sun<sup>d</sup>, Li Qihan<sup>d</sup>, Sei-ichiro Kamata<sup>d</sup>, Edward Ho<sup>f</sup>, Edward Wang<sup>f</sup>, Asaanth Sivajohan<sup>f</sup>, Saerom Youn<sup>f</sup>, Kevin Lane<sup>f</sup>, Jin Chun<sup>f</sup>, Xinliang Wang<sup>g</sup>, Yunchao Gu<sup>g</sup>, Sixu Lu<sup>h</sup>, Young-tack Oh<sup>i</sup>, Hyunjin Park<sup>j,k</sup>, Chia-Yen Lee<sup>l</sup>, Hung Yeh<sup>m,1</sup>, Kai-Wen Cheng<sup>l</sup>, Haoyu Wang<sup>n</sup>, Jin Ye<sup>o</sup>, Junjun He<sup>n,o</sup>, Lixu Gu<sup>n</sup>, Dominik Müller<sup>P,q</sup>, Iñaki Soto-Rey<sup>P,q</sup>, Frank Kramer<sup>P</sup>, Hidehisa Arai<sup>r</sup>, Yuma Ochi<sup>s</sup>, Takami Okada<sup>t</sup>, Luca Giancardo<sup>u,1</sup>, Gwenolé Quellec<sup>v,1</sup>, Fabrice Mériaureau<sup>w,1</sup>

ARTICLE INFO

Article history:

Received –

Received in final form –

Accepted –

Available online –

Communicated by –

Keywords: retinal fundus images, rare pathology detection, ocular disease, classification, multi-label classification

ABSTRACT

In the last decades, many publicly available large fundus image datasets have been collected for diabetic retinopathy, glaucoma, and age-related macular degeneration, and a few other frequent pathologies. These publicly available datasets were used to develop a computer-aided disease diagnosis system by training deep learning models to detect these frequent pathologies. One challenge limiting the adoption of a such system by the ophthalmologist is, computer-aided disease diagnosis system ignores sight-threatening rare pathologies such as central retinal artery occlusion or anterior ischemic optic neuropathy and others that ophthalmologists currently detect. Aiming to advance the state-of-the-art in automatic ocular disease classification of frequent diseases along with the rare pathologies, a grand challenge on “Retinal Image Analysis for multi-Disease Detection” was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI - 2021). This paper, reports the challenge organization, dataset, top-performing participants solutions, evaluation measures, and results based on a new “Retinal Fundus Multi-disease Image Dataset” (RFMiD). There were two principal sub-challenges: disease screening (i.e. presence versus absence of pathology - a binary classification problem) and disease/pathology classification (a 28-class multi-label classification problem). It received a positive response from the scientific community with 74 submissions by individuals/teams that effectively entered in this challenge. The top-performing methodologies utilized a blend of data-preprocessing, data augmentation, pre-trained model, and model ensembling. This multi-disease (frequent and rare pathologies) detection will enable the development of generalizable models for screening the retina, unlike the previous efforts that focused on the detection of specific diseases.

© 2024 Elsevier B. V. All rights reserved.

<footnote><sup>a</sup>Center of Excellence in Signal and Image Processing, Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded 431606, India</footnote>

<footnote><sup>b</sup>Eye Clinic, Sushrusha Hospital, Nanded 431601, India</footnote>

<footnote><sup>c</sup>Department of Ophthalmology, Shankarrao Chavan Government Medical College, Nanded 431606, India</footnote>

<footnote><sup>d</sup>Graudate School of Information Production and Systems, Waseda University, Japan</footnote>

<footnote><sup>e</sup>University of Shanghai for Science and Technology, Shanghai, China</footnote>

<footnote><sup>f</sup>Schulich Applied Computing in Medicine, University of Western Ontario, Schulich School of Medicine and Dentistry</footnote>

<footnote><sup>g</sup>Beihang University School of computer science</footnote>

<footnote><sup>h</sup>Beijing Normal University School of artificial intelligence</footnote>

<footnote><sup>i</sup>Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea</footnote>

<footnote><sup>j</sup>Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Korea</footnote>

<footnote><sup>k</sup>School of Electronic and Electrical Engineering, Sungkyunkwan University, Suwon, Korea</footnote>

<footnote><sup>l</sup>Department of Electrical Engineering, National United University, Miaoli 360001, Taiwan(R.O.C)</footnote>

<footnote><sup>m</sup>Institute of Biomedical Engineering, National Yang Ming Chiao Tung University, 1001 Ta-Hsueh Road Hsinchu, Taiwan(R.O.C)</footnote>

<footnote><sup>n</sup>School of Biomedical Engineering, the Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China</footnote>

<footnote><sup>o</sup>ShenZhen Key Lab of Computer Vision and Pattern Recognition, Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China</footnote>

<footnote><sup>p</sup>IT-Infrastructure for Translational Medical Research, University of Augsburg, Germany</footnote>

<footnote><sup>q</sup>Medical Data Integration Center, University Hospital Augsburg, Germany</footnote>

<footnote><sup>r</sup>NABLAS Inc</footnote>

<footnote><sup>s</sup>National Institute of Technology, Kisarazu College, Japan</footnote>

<footnote><sup>t</sup>Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health</footnote>

<footnote><sup>u</sup>Center for Precision Health, School of Biomedical Informatics, University of Texas Health Science Center at Houston (UTHealth), Houston, TX 77030, USA</footnote>

<footnote><sup>v</sup>Inserm, UMR 1101, F-29200 Brest, France</footnote>

<footnote><sup>w</sup>ImViA EA 7535/IFTIM, Université de Bourgogne, 21078 Dijon, France</footnote>
# 1. Introduction

Eye conditions are prevalent, and on the rise. According to the world health organization (WHO), world report on vision 2019, at present, the total number of people around the world with vision impairment is estimated to be 2.2 billion, from which at least 1 billion vision impairment cases could have been prevented or are yet to be addressed. These over 1 billion people are living with treatable or preventable conditions, just because they can't get the eye care they need. The global demand for eye care is set to swell in the coming years because of rapid population growth, aging, and lifestyle changes. Vision impairment also causes an immense global financial burden due to productivity loss (WHO, 2019).

Color fundus retinal photography uses a camera with a specialized low-powered microscope to non-invasively capture the interior surface of the eye. Retinal fundus images are widely used for clinical screening, examining the presence of disorders, and monitor their change over time. The world faces considerable challenges in terms of eye care, including inequalities in the coverage and quality of prevention, treatment, and rehabilitation services. Detailed clinical observations of characteristic retinal fundus features—not only provide information about eye diseases—but also have led to the identification of early indicators of a diverse range of long-term conditions, such as diabetes, stroke, hypertension, arteriosclerosis, cardiovascular, neurodegenerative, and renal and fatty liver diseases (MacGillivray et al., 2014; Yang et al., 2015; Chang et al., 2016). Hence, the retina is a potential surrogate organ whose assessments are significant not just for the health of the eye but also to decide whether there are systemic issues. Thus, vision loss and damage to other parts of the body can be avoided by eye screening coupled with timely detection and treatment.

Advances in automated retinal image analysis can be used for early detection and diagnosis of ocular pathologies, which would enable the forestall of visual impairment, reduce workload associated with manual grading by ophthalmologists and optometrists, and save the cost of diagnosis and time. As the ratio of ophthalmologists and optometrists to the population is approximately 1:10,000 in most developed countries. However, the ratio is 1:600,000 in developing countries and millions of people per ophthalmologist in rural areas (OAM and Resnikoff, 2002). This inadequacy of professionals is the primary reason for high rates of vision issues in developing countries. Thus, switching to automated computerized screening systems will help in the detection of main components of the retina, classify the normal, abnormal, and unknown retinal images, which in turn significantly increase the rates of disease detection and treatment in the initial stage of its progression.

Recent scientific and technological advances in computing capacity and artificial intelligence approaches provide a new ability for biomedical scientists to meet the need of clinical

practice (Shortliffe and Blois, 2006; Patton et al., 2006). To meet this requirement, raw retinal images along with their associated ground truths play an important role by providing a common ground for the development, validation, and comparison of techniques to the research community. For multi-disease detection, the ground truths are labels given to the particular image based on the clinical records and visual fields. This need has driven the development and sharing of the retinal image databases by the research groups. In the last few decades, some datasets representing diabetic patients collected through diabetic retinopathy (DR) screening programs are IDRiD (Porwal et al., 2020, 2018a), Messidor (Decencière et al., 2014), and EyePACS (Cuadros and Bresnick, 2009). Also, datasets representing the sight-threatening pathologies such as Glaucoma are Drishti-GS (Sivaswamy et al., 2015), RIM-ONE (Fumero et al., 2011), and Refuge (Orlando et al., 2020), and representing age-related macular degeneration (AMD) are ARIA (Farnell et al., 2008), STARE (Hoover et al., 2000), and (Group et al., 2001). The patients visiting the DR screening program may also suffer from other diseases, such as Glaucoma and AMD; thus, multi-disease detection is crucial. There are few datasets that can be used for multi-disease detection one such dataset is ODIR challenge ODIR-2019 (2019) which is publicly available. In ODIR dataset detailed anomaly labeling is only at patient level, not eye level. Also, it only focused on 8 labels (most anomalies being grouped in the 'other' category for evaluation). These multi-disease detection datasets are either private or incorporate only the most common pathologies. However, detection and diagnosis of other sight-threatening rare pathologies, like central retinal artery occlusion or anterior ischemic optic neuropathy and others is also essential. Due to the unavailability of datasets representing rare pathologies, the methodologies for automated detection of rare pathologies are not developed. Thus, rare pathologies are ignored by the automated screening system that usually occurs in routine clinical checkups. This restricts the adoption of automated screening systems by ophthalmologists.

To address these issues, we introduced a new dataset called “Retinal Fundus Multi-Disease Image Dataset” (RFMiD) (Pachade et al., 2021b). Further, it was used as a base dataset for the organization of the grand challenge “Retinal Image Analysis for multi-Disease Detection” (RIADD) in conjunction with International Symposium on Biomedical Imaging (ISBI - 2021). The RFMiD dataset provides 3200 color fundus images with image-level labels representing a normal or abnormal class. It also provides labels for 45 different diseases/abnormalities annotated by the retinal experts. This challenge brought biomedical and computer vision researchers together, intending to develop generalizable techniques/networks for screening retina, unlike the previous efforts that focused on the detection/classification of specific diseases. Initially, for the development of the algorithm, participants were provided with a training dataset and the ground truth. Later, the individuals/teams to participate in the final phase of the challenge were shortlisted based on the performance of their algorithm on the validation dataset. Finally, the winners were judged based on the test dataset performance. The performance of the algorithm

<footnote>*Corresponding author
e-mail: 2017pec601@sggs.ac.in (Samiksha Pachade)</footnote>

<footnote n="1">¹These authors co-organized the challenge. All others contributed results of their algorithm(s) presented in the paper</footnote>
was estimated by how closely the results of the algorithm match the ground truth. The challenge was divided into two sub-tasks: disease screening and disease/pathology classification.

This challenge paper offers unique contributions. Here are key points highlighting its contributions:

1. Focus on Rare Pathologies: Our challenge uniquely addresses the underdeveloped methodologies for automated detection of rare pathologies by providing datasets representing a broader spectrum of conditions, filling a crucial gap in the literature.

2. Enhancing Generalizability: Our challenge promotes solutions applicable to a broader range of conditions, fostering the development of versatile and robust screening systems, in contrast to previous efforts focusing on specific diseases.

3. Addressing Data Challenges: The challenge tackles issues related to the problem of multi-disease detection, limited data, and class imbalance. It provides insights into handling these challenges through a substantial number of practical tips, contributing valuable knowledge for researchers and practitioners in the field.

4. Development of New Solutions: The challenge has led to the development of a variety of new solutions for multi-disease detection. This outcome highlights the effectiveness of the collaborative efforts and the significance of the challenge in advancing the state-of-the-art in the field.

The rest of the paper is organized as follows: section 2 provides a short review of retinal image analysis for multi-disease detection, section 3 gives details of the RFMiD dataset, section 4 provides the organization of the competition, section 5 describes the summary of top-performing participants solutions, section 6 describes evaluation measures used for performance evaluation in this challenge, and section 7 presents the result and comparison of participating teams for both sub-challenges, section 8 gives a brief discussion and conclusion.

## 2. Review of Challenges and Retinal Image Analysis for Multi-disease Detection

In the past decade, few challenges are organized for retinal image analysis, for DR (IDRiD Porwal et al. (2018a), DRIVE Staal et al. (2004), ROC Niemeijer et al. (2009), Kaggle DR detection Cuadros and Bresnick (2009), DeepDRiD Liu et al. (2022)), Glaucoma (REFUGE Orlando et al. (2020), AIROGS de Vente et al. (2021)), AMD (ADAM Fu et al. (2020)), pathological myopia (PM) (PALM PALM-2019 (2019)), and multipathology (ODIR ODIR-2019 (2019)). Such challenges are well known to enable advances in the field by promoting participation of scientific research community from all over the world in a competition at the same time for scientific advancement. IDRiD challenge Porwal et al. (2020, 2018a) dataset consists of 516 retinal images with information on disease severity of DR, diabetic macular edema, and normal retinal structures (optic disc and fovea center coordinates) for each image. Also, it provides 81 images with typical DR lesions (microaneurysms, hemorrhages, hard exudates, and soft exudates) and

optic disc annotated at a pixel level. DRIVE Challenge Staal et al. (2004) dataset consists of 40 images with pixel-level vessel ground truth for all images marked by three experts. ROC Challenge Niemeijer et al. (2009) dataset consists of 100 fundus images for detection of microaneurysms (primary signs of DR) which are marked by four experts. Kaggle DR detection Challenge Cuadros and Bresnick (2009) dataset consists of 88,702 retinal color fundus images. The dataset provides severity grading for DR labeled by two experts. DeepDRiD challenge Liu et al. (2022) dataset composed of 2,000 regular DR images and 256 ultra-widefield images, both having DR quality and grading annotations. REFUGE Challenge Orlando et al. (2020) dataset consists of total 1200 retinal fundus images. The dataset provides ground truth for OD and OC segmentation, fovea localization, and classification of clinical glaucoma annotated by seven independent retinal experts. AIROGS challenge de Vente et al. (2021) dataset contains 113,893 color fundus images with the labels referable glaucoma, no referable glaucoma, or ungradable. ADAM challenge Fu et al. (2020) dataset consists of 1200 images with annotations for diagnosis of AMD, detection and segmentation of optic disc, fovea localization, and segmentation of lesions in fundus photos from AMD patients. The images are annotated by seven independent retinal experts. PALM Challenge PALM-2019 (2019) dataset consists of 1200 images with ground truth for OD segmentation and localization, fovea localization, lesion (atrophy and detachment) segmentation, and classification of PM and non-PM fundus images annotated by seven independent experts. ODIR challenge ODIR-2019 (2019) dataset consists of 5000 images. The images were given 8 different labels namely, normal, diabetes, glaucoma, cataract, AMD, hypertension, myopia, and other diseases/abnormalities.

Many previous studies have focused on retinal disease detection and image analysis using machine learning approaches. Several machine learning algorithms such as the k-nearest neighbor (kNN) algorithm, artificial neural network (ANN), Naive Bayes classifier, and support vector machine (SVM) have been used for retinal disease detection (Mookiah et al., 2013). For retinopathy lesions classification SVM, Gaussian mixture model (GMM), kNN, and AdaBoost are utilized (Roychowdhury et al., 2013).

With the recent success of deep learning, numerous automatic screening systems for fundus images were recently proposed. Historically most targeted pathologies by those systems are DR, AMD, and glaucoma. This is because large datasets of fundus images representing these diseases have been collected over the past decades. Automatic screening system were proposed for DR (Abràmoff et al. (2016), Gulshan et al. (2016), Raju et al. (2017), Gargeya and Leng (2017), Mansour (2018), Nielsen et al. (2019), Quellec et al. (2019)), glaucoma (Li et al. (2018), Shibata et al. (2018), Li et al. (2018), Christopher et al. (2018), Raghavendra et al. (2018), Ahn et al. (2018), Phan et al. (2019), Diaz-Pinto et al. (2019), Pachade et al. (2021a)), and AMD (Grassmann et al. (2018), Tan et al. (2018), Pead et al. (2019), Peng et al. (2020)). Automated frameworks for the detection of other pathologies such as retinopathy of prematurity (Wang et al. (2018), Hu et al. (2018a)) were also developed.
In addition to DR detection Porwal et al. (2018b) proposed a technique for AMD detection. Similarly, Ting et al. (2017) proposed a method for AMD and glaucoma detection along with DR detection.

Recent studies highlight the importance of efforts in the direction of multi-disease detection. Choi et al. (2017) applied a convolutional neural network (CNN) on fundus images for multi-categorical disease detection. VGG-19 architecture with random forest transfer learning gave the most favorable outcomes. The performance of the deep learning models were diminished with the increase in the number of categories. Keel et al. (2019) presented a heat map visualization network using CNNs for the detection of referable DR and glaucomatous optic neuropathy. Son et al. (2020) presented deep learning algorithms that detect the presence of multiple abnormal findings and show lesion heat maps of multiple abnormal findings in retinal fundus photographs. Quellec et al. (2020) first time presented the automatic detection of a large number of (frequent and rare) conditions in retinal images. A new few-shot learning framework was proposed for this purpose. Initially, for the detection of frequent conditions, a deep learning classifier is trained and then for rare condition detection, unsupervised probabilistic models are obtained from these deep learning models.

# 3. Retinal Fundus Multi-disease Image Dataset

## 3.1. Data Acquisition

The retinal images were acquired using three different digital fundus cameras, TOPCON 3D OCT-2000, Kowa VX-10α, and TOPCON TRC-NW300 during the period 2009–2020. They are centered either on the macula or the optic disc (OD). These images are acquired at an Eye Clinic, Sushrusha Hospital, and Center of Excellence in Signal and Image Processing, SGGS Institute of Engineering and Technology both located in Nanded, (M.S.), India. The fundus images are collected from subjects visiting due to a concern for their eye health. Prior to image capturing, the pupils of most of the patients were subjected to mydriasis with one drop of tropicamide at 0.5% concentration. Fundus image acquisition is done with the position and orientation of the subjects sitting upright with 39 mm (Kowa VX-10) and 40.7 mm (TOPCON 3D OCT-2000 and TOPCON TRC-NW300) distance between lenses and examined eye. The details of the camera model, field of view (FOV), resolution, and number of images included in the dataset are given in table 1. The dataset provides expert disease annotation.

<table><caption>Table 1. The details of the camera model, field of view (FOV), resolution, and number of images included in the dataset.</caption><thead><tr><td>Model</td><td>FOV</td><td>Resolution (in pixels)</td><td>Number of images in dataset</td></tr></thead><tbody><tr><td>TOPCON 3D OCT - 2000</td><td>45°</td><td>2144 × 1424</td><td>2427</td></tr><tr><td>Kowa VX - 10α</td><td>50°</td><td>4288 × 2848</td><td>467</td></tr><tr><td>TOPCON TRC-NW300</td><td>45°</td><td>2048 × 1536</td><td>306</td></tr></tbody></table>

## 3.2. Annotation of Images

It comprises information that represents the overall risk factor based on the assessment of the entire image. Two ophthalmologists independently graded the full set of 3200 fundus images. They used a thorough assessment of patients' clinical records and visual fields as the reference standard for the presence of various diseases. The images depict the presence of no disease, single disease, or more than one disease. If it shows the presence of more than one disease, then multiple labels are assigned to the single image. After completion of the initial labeling by both the ophthalmologists, the leader of the project team checked and confirmed or corrected through consultation from both the ophthalmologists when the difference in diagnostic assessments was observed to get adjudicated consensus for the labels. Ground truth for all the fundus images is available in the CSV file.

A brief explanation of the disease/abnormality with their significant visual characteristics of all 46 categories (1 normal and 45 diseases/pathologies) is given in our data descriptor paper (Pachade et al., 2021b). (https://doi.org/10.3390/data6020014).

# 4. Challenge Organization

To boost the efforts towards the development of a generalizable retinal screening system, this dataset is available as a part of RIADD challenge organized in conjunction with IEEE ISBI-2021, Nice, France.

The RIADD challenge was composed of various stages. Figure 1 shows the overall challenge organization workflow. The challenge was officially announced on 16th October 2020 at ISBI-2021 website (http://biomedicalimaging.org/2021/challenges/). The challenge is divided into two sub-challenges as follows:

1. Screening of retinal images into normal and abnormal (comprising of 45 different types of diseases/pathologies) categories.

2. Classification of retinal images into 28 different categories.

For sub-challenge 2, the diseases having more than 10 images belong to an independent class and all other disease categories were merged and labeled as “OTHER”. This finally constitute 28 diseases/pathologies groups for disease classification.

The challenge is comprised of 4 stages, given as follows:

## 4.1. Stage 1: Data Preparation and Distribution

The RFMiD dataset is formed by extracting 3200 images from the thousands of examinations. Both high-quality and low-quality images are selected to make the dataset challenging. The dataset is split into 3 subsets for training 60% (1920 images), evaluation set 20% (640 images), and test set 20% (640 images). The disease-wise stratification on average in training, evaluation, and test set is 60% ± 7%, 20% ± 7% and 20% ± 5% respectively. The detailed dataset breakdown is shown in table 2.
<figure><img src="image_20.png" /><figcaption>Fig. 1. Workflow of the ISBI - 2021: Retinal Image Analysis for multi-Disease Detection (RIADD) Challenge</figcaption></figure>

<table><caption>Table 2. Summary of data augmentation, normalization and pre-processing in the competing solutions. Where, RF, RR, RS, RT, RC represent random flip, rotation, scaling, translation and crop respectively.</caption><thead><tr><th>Sr. No.</th><th>Normal/Disease/Marker</th><th># of Training Set Images</th><th># of Validation Set Images</th><th># of Test Set Images</th><th>Sr. No.</th><th>Normal/Disease/Marker</th><th># of Training Set Images</th><th># of Validation Set Images</th><th># of Test Set Images</th></tr></thead><tbody><tr><td>1</td><td>NL</td><td>401</td><td>134</td><td>134</td><td>24</td><td>CRS</td><td>32</td><td>11</td><td>11</td></tr><tr><td>2</td><td>DR</td><td>376</td><td>132</td><td>134</td><td>25</td><td>EDN</td><td>15</td><td>5</td><td>4</td></tr><tr><td>3</td><td>ARMD</td><td>100</td><td>38</td><td>31</td><td>26</td><td>RPEC</td><td>22</td><td>6</td><td>4</td></tr><tr><td>4</td><td>MH</td><td>317</td><td>102</td><td>104</td><td>27</td><td>MHL</td><td>11</td><td>3</td><td>3</td></tr><tr><td>5</td><td>DN</td><td>138</td><td>46</td><td>46</td><td>28</td><td>RP</td><td>6</td><td>2</td><td>2</td></tr><tr><td>6</td><td>MYA</td><td>101</td><td>34</td><td>32</td><td>29</td><td>CWS</td><td>3</td><td>3</td><td>2</td></tr><tr><td>7</td><td>BRVO</td><td>73</td><td>23</td><td>23</td><td>30</td><td>CB</td><td>1</td><td>0</td><td>1</td></tr><tr><td>8</td><td>TSLN</td><td>186</td><td>65</td><td>53</td><td>31</td><td>ODPM</td><td>0</td><td>2</td><td>0</td></tr><tr><td>9</td><td>ERM</td><td>14</td><td>7</td><td>5</td><td>32</td><td>PRH</td><td>2</td><td>1</td><td>2</td></tr><tr><td>10</td><td>LS</td><td>47</td><td>17</td><td>15</td><td>33</td><td>MNF</td><td>3</td><td>0</td><td>0</td></tr><tr><td>11</td><td>MS</td><td>15</td><td>5</td><td>7</td><td>34</td><td>HR</td><td>0</td><td>0</td><td>1</td></tr><tr><td>12</td><td>CSR</td><td>37</td><td>11</td><td>13</td><td>35</td><td>CRAO</td><td>2</td><td>2</td><td>0</td></tr><tr><td>13</td><td>ODC</td><td>282</td><td>72</td><td>91</td><td>36</td><td>TD</td><td>3</td><td>2</td><td>4</td></tr><tr><td>14</td><td>CRVO</td><td>28</td><td>8</td><td>9</td><td>37</td><td>CME</td><td>4</td><td>1</td><td>2</td></tr><tr><td>15</td><td>TV</td><td>6</td><td>2</td><td>2</td><td>38</td><td>PTCR</td><td>5</td><td>1</td><td>0</td></tr><tr><td>16</td><td>AH</td><td>16</td><td>4</td><td>5</td><td>39</td><td>CF</td><td>3</td><td>2</td><td>1</td></tr><tr><td>17</td><td>ODP</td><td>65</td><td>26</td><td>24</td><td>40</td><td>VH</td><td>1</td><td>2</td><td>1</td></tr><tr><td>18</td><td>ODE</td><td>58</td><td>21</td><td>17</td><td>41</td><td>MCA</td><td>1</td><td>0</td><td>0</td></tr><tr><td>19</td><td>ST</td><td>5</td><td>4</td><td>2</td><td>42</td><td>VS</td><td>1</td><td>3</td><td>0</td></tr><tr><td>20</td><td>AION</td><td>17</td><td>5</td><td>4</td><td>43</td><td>BRAO</td><td>2</td><td>2</td><td>0</td></tr><tr><td>21</td><td>PT</td><td>11</td><td>2</td><td>4</td><td>44</td><td>PLQ</td><td>1</td><td>1</td><td>0</td></tr><tr><td>22</td><td>RT</td><td>14</td><td>6</td><td>5</td><td>45</td><td>HPED</td><td>1</td><td>0</td><td>0</td></tr><tr><td>23</td><td>RS</td><td>43</td><td>14</td><td>14</td><td>46</td><td>CL</td><td>1</td><td>0</td><td>1</td></tr></tbody></table>

NL: normal, DR: diabetic retinopathy, ARMD: age-related macular degeneration, MZ: media haze, DN: drusen, MYA: myopia, BRVO: branch retinal vein occlusion, TSLN: tessellation, ERM: epiretinal membrane, LS: laser scar, MS: macular scar, CSR: central serous retinopathy, ODC: optic disc cupping, CRVO: central retinal vein occlusion, TV: tortuous vessels, AH: asteroid hyalosis, ODP: optic disc pallor, ODE: optic disc edema, ST: shunt, AION: anterior ischemic optic neuropathy, PT: parafoveal telangiectasia, RT: retinal traction, RS: retinitis, CRS: chorioretinitis, EDN: exudation, RPEC: retinal pigment epithelium changes, MHL: macular hole, RP: retinitis pigmentosa, CWS: cotton wool spots, CB: coloboma, ODPM: optic disc pit maculopathy, PRH: preretinal hemorrhage, MNF: myelinated nerve fibers, HR: hemorrhagic retinopathy, CRAO: central retinal artery occlusion, TD: tilted disc, CME: cystoid macular edema, PTCR: post traumatic choroidal rupture, CF: choroidal folds, VH: vitreous hemorrhage, MCA: macroaneurysm, VS: vasculitis, BRAO: branch retinal artery occlusion, PLQ: plaque, HPED: hemorrhagic pigment epithelial detachment, CL: collateral

The challenge was hosted on Grand Challenges in Biomedical Imaging Platform (https://grand-challenge.org/), one of the popular platforms for biomedical imaging-related competitions. A challenge website was launched on 1st November 2020 to disseminate challenge-related information. It was also used for registration, data distribution, submission of results and papers, and communication between the organizers and participants.

## 4.2. Stage 2: Registration and Training Data Release

The challenge registration for consideration to ISBI-2021 on-site contest was made open from 8th November 2020 to 28th November 2021.

February 2021. Individual researchers and research teams registered through the challenge website. The training dataset (images and ground truth) was made available on 22nd November 2020. Participants downloaded the dataset and developed or modified their methods. Further, the participants were allowed to use other datasets for the development of a method that was submitted to the challenge, provided that the datasets are publicly available and clearly stated in their submitted paper. Also, the teams were allowed to do private annotations on public data for some specific tasks and then train their models. If they do so, teams need to release their data and annotations to the public through the challenge website before 31st December 2020.
Only one team used other publicly available dataset for training the network without any annotations i.e. in semi-supervised manner. Thus, no data and annotations were released by any team.

### 4.3. Stage 3: Validation Data Release

The validation dataset (only images) was made available on 29th November 2020. Submissions were sought for either of the two different tasks. The result evaluation site was made open for submission from 1st December 2020. All participants were allowed to make up to 5 result submissions per day. The participants could submit their results by 28th February 2021 and a paper describing the approach till 5th March 2021 to the organizers. Results for both sub-challenge were asked to be submitted in a single CSV file, if participants choose to participate in only one sub-challenge, then they are expected to fill other fields with zeros. A system for automatic evaluation of results was set and the performance is displayed on the evaluation leaderboard on the challenge website. The final rank on the leaderboard is determined by using performance in both sub-challenges as follows:

$$ \text{Final Score} = 0.5*(\text{Sub-challenge-1 Score}) + 0.5*(\text{Sub-challenge-2 Score}) $$

The 8 top-performing participating teams and individuals (decided by the experts based on the performance on the evaluation dataset and their method) were invited to participate in the on-site challenge.

### 4.4. Stage 4: Test Data Release and ISBI-2021 Challenge Event

The main challenge event was held in conjunction with the IEEE ISBI-2021, on 13th April 2021 through a virtual platform. The test dataset was made available on the challenge website (online mode). The participants were given one hour to work on the test data and submit their results. Further, the top three teams from both sub-challenges presented their approach. Finally, the summary of the challenge and final ranking of the competing solutions were presented.

## 5. Summary of Competing Solutions

In this section, we discuss the competition solutions of top performing 8 teams in terms of the following four sections: data pre-processing, data augmentation, pre-trained model, and deep learning models used for classification. The details of competing solutions of each teams are given in Appendix A.

### 5.1. Data Pre-processing

The outcome of any algorithm to solve any particular problem is heavily dependent on the proper distributed data. Thus, data pre-processing is a required first step in designing the network. We analyzed different pre-processing steps used by different teams for the disease screening and classification tasks. One of the pre-processing steps, region of interest (ROI) extraction method that cuts out the redundant black edge from fundus images was used by all 8 teams. No team adopted any background subtraction approach. A summary of the pre-processing steps of all the teams is given in table 3.

<table><caption>Table 3. Summary of data pre-processing used by all the top performing teams. Where ROI is extraction which cuts out the redundant black edge from fundus images; Resize is if the images are resized before giving to the network; CLAHE is if contrast limited adaptive histogram equalization (CLAHE) is applied to images; Green is if the green channel is extracted and given to the network.</caption><thead><tr><th>Team Name</th><th>ROI</th><th>Resize</th><th>CLAHE</th><th>Green</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>Schulich Applied Computing inMed</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>BNUAA</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>Nekaravuru</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>WWW</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>IGSTfencing</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>MISIT</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr><tr><td>chizu & arai & okada</td><td>✓</td><td>✓</td><td>×</td><td>×</td></tr></tbody></table>

### 5.2. Data Augmentation and Normalization

Data augmentation is the set of techniques that are used to artificially increase the size of the data by creating modified copies of images from the already existing dataset, which in turn prevents data scarcity. It is used to increase the generalizability of the data model by reducing data overfitting. It also helps in resolving the class imbalance issue in the classification. Most teams used data augmentation methods, such as flipping, rotation, cropping, and so on, to maintain the models generality. Along with these general augmentations, additional color adjustment methods, i.e., Cutout (DeVries and Taylor, 2017) and Mixup (Zhang et al., 2017) are used. While training the network data normalization makes the model converge faster. As most of the teams have used ImageNet pre-trained model, ImageNet normalization for each of the three RGB channel using mean = [0.485, 0.456, 0.406] and standard deviations = [0.229, 0.224, 0.225]. These values are calculated using millions of images from the ImageNet dataset. The summary of the data augmentation and whether ImageNet normalization is used for all the teams is given in table 4.

### 5.3. Deep Learning Models

Using a pre-trained model requires less effort in building the model's architecture, requires less training, and has proven to be accurate with proper fine-tuning as compared to a model using a custom-built CNN, especially when the number of annotated samples are quite low. In this challenge, the ImageNet pre-training model was used by most teams to improve their model performance. A summary of the pre-training model is given in table 5.

In this challenge, all teams used deep learning models to construct classification frameworks. Most teams adopted Efficient-Net (Tan and Le, 2019) as their deep learning backbones, and achieved excellent classification performance, whereas some teams selected classical ResNet (He et al., 2016), ResNext (Xie et al., 2017), InceptionV3 (Szegedy et al., 2016), DenseNet (Huang et al., 2017), SE-ResNeXt (Hu et al., 2018b), TRes-Net (Ridnik et al., 2021), SENet (Hu et al., 2018c) and NFNet (Brock et al., 2021). Team IGSTfencing used an external public dataset without labels for training the semi-supervised model. Most teams chose binary cross-entropy (BCE) loss function. All the teams used ensemble models which combine multiple
<table><caption>Table 4. Summary of data augmentation and normalization used by all the top performing teams. Where RF, RR, HSV, RB/RBC, RC, RRC, and SSR represents random flip, random rotation, hue saturation value, random brightness/ random brightness contrast, random crop, random resized crop, and shift scale rotate respectively.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="10">Augmentation</th><th rowspan="2">ImageNet normalization</th></tr><tr><th>RF</th><th>RR</th><th>HSV</th><th>RB/RBC</th><th>RC</th><th>RRC</th><th>SSR</th><th>Cutout</th><th>Mixup</th><th>Other</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>median blur, additive Gaussian noise</td><td>✓</td></tr><tr><td>Schulich Applied Computing inMed</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✗</td></tr><tr><td>BNUAA</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td><td>✓</td><td>resize, transpose, coarse dropout, random scale, shifting, blur, elastic transformations, grid distortion</td><td>✗</td></tr><tr><td>Nekaravuru</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>–</td><td>✗</td></tr><tr><td>WWW</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✗</td></tr><tr><td>IGSTfencing</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✓</td></tr><tr><td>MISIT</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>–</td><td>✓</td></tr><tr><td>chizu & arai & okada</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>–</td><td>✗</td></tr></tbody></table>

<table><caption>Table 5. Summary of pre-training model used and deep learning model with loss functions and model ensembling. Where BCE is binary cross-entropy, CE is cross-entropy, SC1 is sub-challenge 1, and SC2 is sub-challenge 2.</caption><thead><tr><th>Team Name</th><th>ImageNet pre-trained mode</th><th>Model framework</th><th>Loss function</th><th>Model ensembling</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>✓</td><td>EfficientNet B5 and B6<br>SE-ResNeXt, InceptionV3,<br>EfficientNet B4 and B5,<br>and DenseNet-121</td><td>BCE with logits</td><td>✓</td></tr><tr><td>Schulich Applied Computing inMed</td><td>✓</td><td>ResNet and ResNext</td><td>asymmetric loss</td><td>✓</td></tr><tr><td>BNUAA</td><td>✗</td><td>EfficientNet B0, B1, and B2</td><td>CE and BCE loss</td><td>✓</td></tr><tr><td>Nekaravuru</td><td>✓</td><td>EfficientNet and TResNet</td><td>weighted BCE loss</td><td>✓</td></tr><tr><td>WWW</td><td>✓</td><td>SS-GCN with SENet154<br>as backbone</td><td>asymmetric loss</td><td>✓</td></tr><tr><td>IGSTfencing</td><td>✓</td><td>SC1: EfficientNet B4 and InceptionV3<br>SC2: ResNet152, InceptionV3,<br>DenseNet-201, and EfficientNet B4</td><td>BCE, multi-view<br>and co-regularization</td><td>✓</td></tr><tr><td>MISIT</td><td>✓</td><td>EfficientNet and NFNet</td><td>weighted focal loss</td><td>✓</td></tr><tr><td>chizu & arai & okada</td><td>✓</td><td></td><td>focal loss</td><td>✓</td></tr></tbody></table>

models in the prediction process. Specifically, the majority of teams opted for the averaging ensembling technique. This approach involves averaging predictions from individual models to derive the final prediction. Notably, team MISIT employed a combination of bagging and stacking ensembling techniques, while team Chizu & Arai & Okada utilized a weighted averaging ensembling technique. Bagging involves training multiple instances of the same base model on distinct subsets of the training data. The predictions from each model in the bagging ensemble are then aggregated through stacking. It's important to highlight that all teams generated multiple predictions using different models, rather than relying on a single model with augmented versions of the same inference image. This ensemble of diverse models significantly contributes to the overall robustness and accuracy of the predictions across all teams. A summary of the deep learning model with loss and model ensemble is given in table 5.

Sub-challenge-1 (disease screening) comprises two categories: normal and abnormal. Sub-challenge-2 (disease classification) comprises 28 different categories. The disease risk label in sub-challenge-1 aligns consistently with multiple labels representing different diseases and pathologies in sub-challenge-2. This indicates that the labels across multiple categories contain sufficient information relevant to both sub-challenge-1 and sub-challenge-2. Thus, the majority of teams approach combined handling of both sub-challenges as a unified task with 29 classes (1 class for sub-challenge-1 and 28 classes for sub-challenge-2). Teams BNUAA and WWW employed a single deep learning model as the base model for addressing both sub-challenges. Furthermore, the data pre-

processing, augmentation techniques, and training details utilized by all the teams are consistent for both sub-challenges.

## 6. Evaluation Measures

For the RIADD challenge, model performance was assessed with two metrics, “Disease Risk Score” and “Multi-Disease Average Score” each corresponding to one of the two sub-challenges: 1. Disease Screening: the ability of the model to detect the presence of any disease. This was measured by the (“Disease Risk Score”) AUC for the predictions of the normal/abnormal label only. 2. Disease Classification: the performance of the model in labelling a given image with the correct pathological findings. This was measured by (“Multi-Disease Average Score”) averaging the mean average precision (mAP) over each label with the average AUC over each label:

$$ \text{Multi-disease Avg Score} = \frac{1}{2} \left( \frac{1}{c} \sum_{i=0}^{c} AUC_i + mAP_i \right) \quad (1) $$

where c is the class index, and mAP is defined as:

$$ mAP = \sum_{n}^{N} (R_{n} - R_{n-1})p_{n} \quad (2) $$

where N is the number of unique threshold values, and $p_n$ and $R_n$ are the precision and recall, respectively, at the n-th threshold.

Additionally, we conducted an analysis for both tasks, evaluating F1 score and Specificity. It's noteworthy that, during
<table><caption>Table 6. The best submission performance of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi-Disease Average Score) on validation set. AUC: area under the receiver operator curve, mAP: mean average precision, SC1: sub-challenge-1 and SC2: sub-challenge-2. Average values are reported for SC2.</caption><thead><tr><th rowspan="2">Team Name</th><th rowspan="2">Disease Risk Score AUC</th><th rowspan="2">Multi-Disease Average Score AUC + mAP</th><th rowspan="2">Final Score</th><th rowspan="2">Rank</th><th colspan="2">SC1</th><th colspan="2">SC2</th></tr><tr><th>F1 Score</th><th>Specificity</th><th>F1 Score</th><th>Specificity</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9702</td><td>0.8017</td><td>0.8859</td><td>1</td><td>0.9411</td><td>0.8783</td><td>0.5561</td><td>0.9935</td></tr><tr><td>IGSTfencing</td><td>0.9601</td><td>0.7911</td><td>0.8756</td><td>2</td><td>0.9409</td><td>0.7612</td><td>0.5433</td><td>0.9917</td></tr><tr><td>chizu & arai & okada</td><td>0.9636</td><td>0.7873</td><td>0.8755</td><td>3</td><td>0.9382</td><td>0.8308</td><td>0.6247</td><td>0.9911</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9613</td><td>0.7858</td><td>0.8735</td><td>4</td><td>0.9335</td><td>0.5896</td><td>0.5467</td><td>0.9847</td></tr><tr><td>BNUAA</td><td>0.9610</td><td>0.7820</td><td>0.8715</td><td>5</td><td>0.9439</td><td>0.8433</td><td>0.5904</td><td>0.9931</td></tr><tr><td>Nekaravuru</td><td>0.9626</td><td>0.7566</td><td>0.8596</td><td>6</td><td>0.9369</td><td>0.8806</td><td>0.5013</td><td>0.9846</td></tr><tr><td>WWW</td><td>0.9572</td><td>0.7584</td><td>0.8578</td><td>7</td><td>0.9339</td><td>0.8060</td><td>0.5213</td><td>0.9886</td></tr><tr><td>MISIT</td><td>0.9490</td><td>0.7027</td><td>0.8258</td><td>8</td><td>0.9261</td><td>0.8731</td><td>0.6614</td><td>0.9900</td></tr></tbody></table>

<table><caption>Table 7. The performance of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi-Disease Average Score) on test set. AUC: area under the receiver operator curve, mAP: mean average precision, SC1: sub-challenge-1 and SC2: sub-challenge-2. Average values are reported for SC2.</caption><thead><tr><th rowspan="2">Team Name</th><th rowspan="2">Disease Risk Score AUC</th><th rowspan="2">Multi-Disease Average Score AUC + mAP</th><th rowspan="2">Final Score</th><th rowspan="2">Rank</th><th colspan="2">SC1</th><th colspan="2">SC2</th></tr><tr><th>F1 Score</th><th>Specificity</th><th>F1 Score</th><th>Specificity</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9875</td><td>0.7821</td><td>0.8848</td><td>1</td><td>0.9625</td><td>0.9668</td><td>0.5546</td><td>0.9957</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9861</td><td>0.7782</td><td>0.8821</td><td>2</td><td>0.9632</td><td>0.7836</td><td>0.5411</td><td>0.9876</td></tr><tr><td>BNUAA</td><td>0.9870</td><td>0.7578</td><td>0.8724</td><td>3</td><td>0.9593</td><td>0.9627</td><td>0.6516</td><td>0.9956</td></tr><tr><td>Nekaravuru</td><td>0.9782</td><td>0.7485</td><td>0.8634</td><td>4</td><td>0.9478</td><td>0.9403</td><td>0.5013</td><td>0.9871</td></tr><tr><td>WWW</td><td>0.9765</td><td>0.7380</td><td>0.8572</td><td>5</td><td>0.9170</td><td>0.9776</td><td>0.5089</td><td>0.9894</td></tr><tr><td>IGSTfencing</td><td>0.9797</td><td>0.7102</td><td>0.8449</td><td>6</td><td>0.9512</td><td>0.9254</td><td>0.5529</td><td>0.9923</td></tr><tr><td>MISIT</td><td>0.9706</td><td>0.6893</td><td>0.8300</td><td>7</td><td>0.9475</td><td>0.7899</td><td>0.6668</td><td>0.9891</td></tr><tr><td>chizu & arai & okada</td><td>0.9885</td><td>0.7796</td><td>0.8840</td><td>*</td><td>0.9506</td><td>0.9701</td><td>0.6445</td><td>0.9968</td></tr></tbody></table>

the challenge, these metrics were not utilized as evaluation parameters and, as a result, were not considered in the ranking of participating teams. The F1 score balances precision and recall, providing a comprehensive measure of a model's ability to correctly identify positive instances while minimizing false positives. Specificity, on the other hand, gauges a model's proficiency in correctly identifying true negatives, offering insights into its ability to avoid false alarms. Together, these metrics contribute to a more nuanced understanding of a model's effectiveness.

## 7. Results

This section reports and discusses the results of the challenge. Performance of all competing solutions on both sub-challenge including their leaderboard rank. The final rank on the leaderboard was determined by using performance in both sub-challenges as follows:

$$ \text{Final Score} = 0.5*(\text{Sub-challenge-1 Score}) + 0.5*(\text{Sub-challenge-2 Score}) $$

This section presents the performance of the proposed model by all participating teams for disease screening and classification tasks. Online result submission and automated evaluation system was built and presented on our website for the participants to submit and evaluate their results. Evaluated results are made publicly available on the leaderboard of our website. All the results submitted by competing teams were analyzed using the evaluation measures given in section 6. This measure generated the AUC score for sub-challenge-1 and AUC and mAP score for sub-challenge-2. We got total 74 submissions by individuals/teams for the challenge. Amongst them, the top performing 8 teams were invited to participate in the final online challenge.

Table 6 summarizes the best submission performance of the top 8 teams on the validation set. The column "Disease Risk Score" shows the performance of the individual solution for disease screening task (sub-challenge 1) and the column "Multi-Disease Average Score" gives the performance for the disease classification task (sub-challenge 2). The column "Final Score" is an aggregate performance of the individual solution on sub-challenge 1 and sub-challenge 2. The final placement for each subtask is determined using the "Final Score" from the leaderboard. Furthermore, we conducted an analysis for both tasks (SC1: sub-challenge-1 and SC2: sub-challenge-2), evaluating F1 score and Specificity. The performance of top 8 teams is reported in the same table. The "Final Score" of all the participating individual/ teams in decreasing order of their performance in sub-challenge-1 and sub-challenge-2 on validation set is shown in Fig.2 and 3 respectively. Fig.4 shows the distribution of "Final Score" in sub-challenge-2 across the top performing eight teams for all 28 disease categories on test set. The disease categories are ranked by decreasing order of their frequency of appearance in the RFMiD dataset.

Table 7 summarizes the individual performance on the test set of the top 8 teams selected based on the performance of their proposed solution on the validation set. Amongst the 8 teams on the day of the challenge, 7 teams submitted their results through the automated online evaluation system. The team chizu & arai & okada could not participate on the day of challenge but later submitted their results and thus they are not ranked. The top-performing solution on the day of challenge was proposed by team KAMATALAB followed by team Schulich Applied Computing inMed and team BNUAA with a final score of 0.8848, 0.8821, and 0.8724, respectively.

To provide a more comprehensive analysis of the top-
<figure><img src="image_21.png" /><figcaption>Fig. 2. The performance (Final Score) of all the participating individual/ teams in sub-challenge-1 on validation set.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Fig. 3. The performance (Final Score) of all the participating individual/ teams in sub-challenge-2 on validation set.</figcaption></figure>

performing results, ranking stability is employed, and boot-
strapping serves as a valuable technique for assessing the ro-
bustness of these rankings. Bootstrapping for ranking stability
is a statistical resampling technique used to assess the reliability
and variability of rankings. It involves repeatedly resampling
the observed data with replacement to create multiple simulated
datasets. Here we estimated 1,000 bootstrap samples. The 95%
confidence interval is determined by calculating the 2.5th and
97.5th percentiles of the distribution of the bootstrapped statis-
tic. This interval represents the range within which the true
population parameter is estimated to fall with 95% confidence
based on the variability observed in the resampled data. The

rankings are then recalculated and by comparing the original
ranking with those obtained from resampled datasets, one can
estimate the stability and uncertainty associated with the rank-
ings.

Tables 8 and 9 present bootstrap confidence intervals (Average, Lower, and Upper) for the best submission performances of the top 8 teams on the validation and test sets, respectively. The average represents the mean of bootstrapped values calculated over 1000 samples. In Table 8, the bootstrapped ranking closely aligns with the original ranking, with only slight alterations observed for ranks 4 (BNUAA) and 5 (Schulich Applied Computing inMed). The final bootstrapped ranking is deter-
<table><caption>Table 8. Bootstrap confidence intervals for best submission performance of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi-Disease Average Score) on validation set. AUC: area under the receiver operator curve, mAP: mean average precision, CI: confidence interval.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="3">Disease Risk Score AUC (95% CI) (Bootstrap)</th><th colspan="3">Multi-Disease Average Score AUC + mAP (95% CI) (Bootstrap)</th><th colspan="3">Final Score (Bootstrap)</th><th rowspan="2">Rank</th></tr><tr><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9724</td><td>0.9591</td><td>0.9827</td><td>0.8043</td><td>0.7821</td><td>0.8223</td><td>0.8884</td><td>0.8675</td><td>0.9151</td><td>1</td></tr><tr><td>IGSTfencing</td><td>0.9599</td><td>0.9456</td><td>0.9728</td><td>0.7981</td><td>0.77804</td><td>0.8171</td><td>0.8790</td><td>0.8673</td><td>0.8900</td><td>2</td></tr><tr><td>chizu & arai & okada</td><td>0.9639</td><td>0.9435</td><td>0.9757</td><td>0.7900</td><td>0.7725</td><td>0.8027</td><td>0.8770</td><td>0.8569</td><td>0.8947</td><td>3</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9611</td><td>0.9476</td><td>0.9740</td><td>0.7914</td><td>0.7718</td><td>0.8108</td><td>0.8763</td><td>0.8637</td><td>0.8878</td><td>5</td></tr><tr><td>BNUAA</td><td>0.9613</td><td>0.9481</td><td>0.9730</td><td>0.79149</td><td>0.7696</td><td>0.8111</td><td>0.8764</td><td>0.8646</td><td>0.8894</td><td>4</td></tr><tr><td>Nekaravuru</td><td>0.9627</td><td>0.9489</td><td>0.9746</td><td>0.76418</td><td>0.7424</td><td>0.7861</td><td>0.8634</td><td>0.8504</td><td>0.8761</td><td>6</td></tr><tr><td>WWW</td><td>0.9572</td><td>0.9425</td><td>0.9696</td><td>0.7627</td><td>0.7409</td><td>0.7841</td><td>0.8600</td><td>0.8469</td><td>0.8728</td><td>7</td></tr><tr><td>MISIT</td><td>0.9489</td><td>0.9316</td><td>0.9640</td><td>0.7085</td><td>0.6822</td><td>0.7354</td><td>0.8287</td><td>0.8120</td><td>0.8451</td><td>8</td></tr></tbody></table>

<table><caption>Table 9. Bootstrap confidence intervals of top 8 teams in sub-challenge 1 (Disease Risk Score) and sub-challenge 2 (Multi- Disease Average Score) on test set. AUC: area under the receiver operator curve, mAP: mean average precision, CI: confidence interval, p-value: performance of Wilcoxon signed-rank tests.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="3">Disease Risk Score AUC (95% CI) (Bootstrap)</th><th colspan="3">Multi-Disease Average Score AUC + mAP (95% CI) (Bootstrap)</th><th colspan="3">Final Score (Bootstrap)</th><th rowspan="2">Rank</th><th rowspan="2">p-value</th></tr><tr><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th><th>Average</th><th>Lower</th><th>Upper</th></tr></thead><tbody><tr><td>KAMATALAB</td><td>0.9880</td><td>0.9789</td><td>0.9928</td><td>0.7888</td><td>0.7676</td><td>0.8047</td><td>0.8885</td><td>0.8688</td><td>0.8947</td><td>1</td><td>0.005</td></tr><tr><td>Schulich Applied Computing inMed</td><td>0.9861</td><td>0.9783</td><td>0.9923</td><td>0.7843</td><td>0.7565</td><td>0.8120</td><td>0.8853</td><td>0.8711</td><td>0.8996</td><td>2</td><td>0.014</td></tr><tr><td>BNUAA</td><td>0.9870</td><td>0.9794</td><td>0.9930</td><td>0.7651</td><td>0.7376</td><td>0.7952</td><td>0.8760</td><td>0.8620</td><td>0.8922</td><td>3</td><td>0.028</td></tr><tr><td>Nekaravuru</td><td>0.9784</td><td>0.9681</td><td>0.9876</td><td>0.7559</td><td>0.7250</td><td>0.7865</td><td>0.8671</td><td>0.8509</td><td>0.8825</td><td>4</td><td>0.056</td></tr><tr><td>WWW</td><td>0.9765</td><td>0.9667</td><td>0.9851</td><td>0.7454</td><td>0.7128</td><td>0.7743</td><td>0.8610</td><td>0.8436</td><td>0.8759</td><td>5</td><td>5.8e-13</td></tr><tr><td>IGSTfencing</td><td>0.9796</td><td>0.9703</td><td>0.9875</td><td>0.7186</td><td>0.6820</td><td>0.7531</td><td>0.8491</td><td>0.8305</td><td>0.8678</td><td>6</td><td>1.2e-07</td></tr><tr><td>MISIT</td><td>0.9236</td><td>0.8965</td><td>0.9474</td><td>0.6635</td><td>0.6309</td><td>0.6968</td><td>0.7936</td><td>0.7727</td><td>0.8142</td><td>7</td><td>-</td></tr><tr><td>chizu & arai & okada</td><td>0.9886</td><td>0.9822</td><td>0.9936</td><td>0.7880</td><td>0.7601</td><td>0.8150</td><td>0.8883</td><td>0.8746</td><td>0.9022</td><td>*</td><td>-</td></tr></tbody></table>

mined based on the overall average value. In Table 9, the bootstrapped ranking precisely mirrors the original ranking.

Kendall's tau (Kendall, 1938) correlation coefficient was employed to measure changes between the original and bootstrap rankings, ranging from 1 to -1. A coefficient of 1 signifies an unchanged ranking, while -1 indicates a reversed ranking (Wagner et al., 2023). The rankings appear stable, with the first and second algorithms closely together. This observation is reinforced by the mean Kendall's tau value across all bootstrap samples of 0.85 (median: 0.91, interquartile range (IQR): (0.81, 0.91)).

Additionally, pairwise Wilcoxon signed rank tests were conducted for multiple comparisons across algorithm pairs. The tests were used to check for algorithms that were significantly superior in terms of performance compared to their competitors (Wagner et al., 2023). The results of this analysis are independent from the bootstrap analysis. Multi-disease detection probabilities computed by two teams on test set data are compared using a Wilcoxon signed-rank test. The resulting p-values obtained for consecutive teams in the team ranking are reported in Table 9. The p-value < 0.05 suggest statistically significant evidence of median differences in performance data between the compared algorithms.

Finally, we tried both Averaging Ensemble and Weighted Averaging Ensemble for the top-performing methods (Table 10), but neither yielded satisfactory results compared to the individual methods. However, it's worth noting that model ensembling, which involves combining the outputs of multiple models, often offers advantages over simple probability averaging methods.

# 8. Discussion and Conclusion

## 8.1. Summary of the Challenge

In this paper, we have presented the details of the RIADD challenge, including detailed information about the dataset, the challenge organization, all the top-performing competing solutions, used evaluation metrics, and final individual and averaged results for both sub-tasks, i.e., disease screening and multi-disease classification.

The top performing methods for multi-disease detection demonstrated that the problem of multi-disease detection, limited data, and class imbalance can be handled using a substantial number of common tips: (1) appropriate data preprocessing, (2) data augmentation to handle the problem of data scarcity and class imbalance problem, (3) transfer learning using the pre-trained model, (4) modification of loss function, (5) model ensembling. Additionally, to improve model performance, many teams used the EfficientNet framework as the backbone model for the network, it learns gradient features by uniformly scaling all dimensions of depth/width/resolution using a compound coefficient. For both the sub-challenges, the best performing proposed network by team "KAMATALAB" along with these common tips, adopted post-processing steps to avoid over-fitting. As this is the first publicly available multi-disease dataset, most of the teams did not use an external dataset. The only team that used external unlabelled dataset is "IGSTfencing", it used ODIR-2019 (ODIR, 2019) dataset and got 2nd and 6th position in sub-challenge 1 and sub-challenge 2, respectively.

Teams are observed (from tables 6 and 7) to perform poorly in the multi-disease classification task as compared to the dis-
<table><caption>Table 10. The performance of Averaging Ensemble and Weighted Averaging Ensemble for the top-performing methods on test set. AUC: area under the receiver operator curve, mAP: mean average precision, M1: KAMATALAB, M2: Schulich Applied Computing inMed, M3: BNUAA.</caption><thead><tr><th rowspan="2">Team Name</th><th colspan="3">Averaging Ensemble</th><th colspan="3">Weighted Averaging Ensemble</th></tr><tr><th>Disease Risk Score AUC</th><th>Multi-Disease Average Score AUC + mAP</th><th>Final Score</th><th>Disease Risk Score AUC</th><th>Multi-Disease Average Score AUC + mAP</th><th>Final Score</th></tr></thead><tbody><tr><td>M1 + M2</td><td>0.9877</td><td>0.7819</td><td>0.8848</td><td>0.9880</td><td>0.7830</td><td>0.8855</td></tr><tr><td>M1 + M2 + M3</td><td>0.9888</td><td>0.7812</td><td>0.8850</td><td>0.9885</td><td>0.7825</td><td>0.8855</td></tr></tbody></table>

<figure><img src="image_23.png" /><figcaption>Fig. 4. Box plot showing the distribution of "Final Score" in sub-challenge-2 across the top performing eight teams for all 28 disease categories on test set. The disease categories are ranked by decreasing order of their frequency of appearance in the RFMiD dataset.</figcaption></figure>

ease screening task that reduced the overall final score. The major reason seems to be the difficulty in rare pathology classification which in turn decreases the overall performance of the multi-disease classification task. The same can also be seen from the box plot (Fig.4), showing the distribution of "Final Score" in sub-challenge-2 across the top performing eight teams for all 28 disease categories on test set. We observe that detection performance generally increases with the size of disease categories. Other factors probably come into play, such as the subtlety of diseases: optic disc pallor (ODP) is frequent but subtle, retinitis pigmentosa (RP) is rare but generally obvious. The 'Other' category, which is both small and varied, is particularly challenging. We note that the 8 teams have similar scores for frequent categories: the performance difference between teams largely comes from rare categories, where the variability between teams is high.

## 8.2. Limitations of the Study

This challenge provided data acquired in routine clinical practice using three different digital fundus cameras, using a consistent acquisition protocol for all images. The images captured with different retinal cameras and with multiple diseases

allowed a better estimation of the generalizability of the proposed architectures. The data was acquired either by ophthalmologists or an expert operator at two different centers. The images without an adjudicated consensus of expert annotator and non-gradable images are not included in the dataset. The period from data availability to the deadline for results submission, about three months, was considerable time for methodology development and submission for all tasks. Further, automated evaluation system have facilitated the submission and evaluation procedure by providing a continuous (real-time feedback) online assessment of participating solutions. This also enabled a maximum number of submissions during the challenge period. Even after these efforts to give the best possible data, adequate time for submission, and an automated evaluation system, the data annotation process is still intrinsically subjective, and the annotators' judgment is a restricting factor for the architecture performance. Further, data was acquired at one location only, for the generalizability of the network, it would be great if the data is from different ethnicity and race.
## 8.3. Future Work

Still, there is room for improvement, especially in multi-disease classification task. Though the competition is now completed, the dataset has been made publicly available for research purposes to attract newcomers to the problem and to encourage the development of novel solutions to meet current and future clinical standards. Additionally, it would be beneficial for the network's generalizability to obtain data from various ethnicities and races, considering that the current data was acquired at a single location.

## 8.4. Conclusion

This challenge not only identifies and addresses critical gaps in existing practices but also promotes collaboration, tackles real-world challenges, and encourages innovative solutions for multi-disease detection in retinal screening. Despite the complexity of the tasks, it received a very positive response and the top performing solutions were proposed by participating teams. Given the significant number of participating teams and the results obtained, we believe this challenge was a success. To the organizational end, efforts have been made in creating an admissible, challenging, and fair competition, capable of facilitating collective knowledge in the research community.

## Declaration of Competing Interest

The authors have no conflicts of interest to declare.

## Acknowledgments

This work is sponsored by the Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded (M.S.), INDIA, French National Research Agency under the LabCom program (ANR-19-LCV2-0005 - ADMIRE project), and OphtAI. The authors would like to thank Dhanashree Thulkar (Veermata Jijabai Technological Institute, Mumbai, India) for their help in various aspects of organizing the ISBI-2021 RIADD Challenge.

# Appendix A. Details of Competing Solutions

## Appendix A.1. KAMATALAB (Luo Zhengbo et al.)

Luo et al. proposed a diverse blending system based on EfficientNet (Tan and Le, 2019) for multi-disease detection as shown in Fig. A.5. It consists of three paths with various image sizes, versions of models, and training strategies (table A.11). The system did not use any extra data, and only a simple pre-processing, which cuts out the black edges, was applied. SGD was used with a Nesterov momentum of 0.9 as an optimizer, the learning rate was set to 0.001 initially and increased to 0.1 shortly, then divided by ten every ten epochs with an early stop (the best epoch usually between 15-20). The direct training strategy refers that: dealing with both sub-challenges together as a 29-classes task (1 class for sub-challenge one and 28 classes for sub-challenge two). The multi-stage strategy refers that: dividing sub-challenge two into three sub-tasks with a similar number of training samples, which are 9 classes

<figure><img src="image_24.png" /><figcaption>Fig. A.5. The overall structure of the proposed system for multi-disease detection.</figcaption></figure>

<table><caption>Table A.11. The details of three paths with various image sizes, version of models, and training strategies.</caption><thead><tr><td></td><td>Version of Model</td><td>Input size (in pixels)</td><td>Training strategy</td></tr></thead><tbody><tr><td>Path one</td><td>Efficient-Net B5</td><td>768 × 768 × 3</td><td>Multi-stage</td></tr><tr><td>Path two</td><td>Efficient-Net B5</td><td>960 × 960 × 3</td><td>Direct training</td></tr><tr><td>Path three</td><td>Efficient-Net B6</td><td>768 × 768 × 3</td><td>Direct training</td></tr></tbody></table>

with over 65 samples, 8 classes with samples number ranging from 20 to 65, and 11 classes under 20 samples. All models are trained through 5-fold cross-validation. The authors adopt both horizontal and vertical flip with $p = 0.5$, median-blur with blur limit 7, additive Gaussian noise which scales in a range of [0, 0.15 * 255], regular hue saturation and brightness augmentations, five holes cutout with max size [20, 20] and ImageNet normalization. However, resizing and imageNet normalization procedure took place for the validation. Test time augmentation (TTA) was applied for inference. Five times TTA for path one and three times TTA for path two and three with setting $p = 0.5$ for horizontal and vertical flip, hue, saturation, random brightness, and ImageNet normalization. The authors applied post-processing mainly on two aspects: minor classes and correlation between two sub-challenges. The post-processing applied are, average blending for major classes only, providing meta-Information, replacement with min-max values, and rank voting. Moreover, the risk score of sub-challenge one depends on whether the image has any disease in sub-challenge two. Decrease predictions with a low-risk score in sub-challenge two gave a reasonable raise in performance. The best off-site result was based on an average blending of all three paths; however, the on-site result was based on path two and three only due to the time limitation. The source code is available at https://github.com/Hanson0910/Pytorch-RIADD.

## Appendix A.2. Schulich Applied ComputingInMed (Edward Ho et al.)

Ho et al. used an ensemble of 5 CNN architectures to predict both disease risk (presence or absence of any pathologies) and disease classification simultaneously in a multilabel, multiclass problem formulation with 29 classes. All models were trained with preprocessed images that were square cropped around the center of the image (TOPCON) or cropped by finding the rectangular area containing all pixels above a constant threshold value (Kowa). The 5 architectures used were SE-ResNeXt (Hu
et al., 2018b), InceptionV3 (Szegedy et al., 2016), EfficientNet-B4 and B5 (Tan and Le, 2019), and DenseNet (Huang et al., 2017). For the SE-ResNeXt model, a dropout layer was added before the last linear layer. All other architectures were un-modified. The hyperparameters and image input sizes for the 5 different architectures are given in table A.12.

<table><caption>Table A.12. Hyperparameters and image input sizes for the 5 different architectures.</caption><thead><tr><th>Architecture</th><th>Image Size (in pixels)</th><th>Batch Size</th><th>Learning Rate</th><th>Training Epoch</th></tr></thead><tbody><tr><td>SE-ResNeXt</td><td>512</td><td>8</td><td>1 × 10<sup>-4</sup></td><td>20</td></tr><tr><td>DenseNet-121</td><td>299</td><td>16</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr><tr><td>InceptionV3</td><td>299</td><td>32</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr><tr><td>EfficientNet-B4</td><td>380</td><td>12</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr><tr><td>EfficientNet-B5</td><td>456</td><td>8</td><td>2 × 10<sup>-4</sup></td><td>10</td></tr></tbody></table>

Networks were all trained by optimizing an asymmetric loss, as described by Ben-Baruch et al. (Ben-Baruch et al., 2020), to counter the significant class imbalances found in this problem. This is a modified BCE loss such that the loss is defined as:

$$L = -yL_{+} - (1 - y)L_{-} \quad (A.1)$$

where $L_+$ and $L_-$ represent the positive and negative loss parts and are defined as:

$$ 
\begin{aligned}
L_{+} &= (1 - p)^{y_{+}} \log(p) \\
L_{-} &= (p_{m})^{y_{-}} \log(1 - p_{m})
\end{aligned} 
\quad (A.2) $$

where $y_+$ and $y_-$ are the positive and negative focusing parameters, respectively; and $p_m$ is the shifted probability defined by:

$$p_m = \max(p - m, 0) \quad (A.3)$$

With the shifted probability in the negative loss part, this allows easy negative examples (such that the prediction probability is less than the probability margin $m$) to not factor into the loss. By setting $y_- > y_+$, the contribution of positive samples to the loss can be increased. The focusing parameters and probability margins are tunable hyperparameters.

All networks were initialized by pre-training on ImageNet and trained using the Adam (Kingma and Ba, 2014). Random horizontal flip and random rotation (between -30 and 30 degrees) augmentations were used for all networks except InceptionV3, which rotated between -90 and 90 degrees and incorporated a random brightness transformation.

For each of the 5 CNN architectures, models, were trained with a 5-fold cross-validation on iteratively stratified folds (Sechidis et al., 2011), (Szymański and Kajdanowicz, 2017) to preserve the distribution of class labels. This results in 5 independent models, and the predicted scores are averaged to serve as the output for a given architecture. Finally, the outputs from all five architectures are averaged to form the final ensemble.

## Appendix A.3. BNUAA (Xinliang Wang et al.)

Wang et al. proposed a novel Disease Head to solve the disease screening and multi-disease detection problems. The authors designed a simple method to binarize the original image

and used the binary image as a mask to delete the invalid background in the original fundus images. In addition, the authors used heavy augmentation resize, transpose, horizontal flip, vertical flip, random brightness contrast, hue saturation value, shift scale rotate, coarse dropout, cut out, and mixup to avoid overfitting. The Ranger optimizer and CosineAnnealingLR learning rate scheduler were employed to achieve faster convergence speed and better results. There are 2 categories in the sub-challenge 1: have fundus diseases or not, and 28 categories in sub-challenge 2: different fundus diseases. The authors found that these two tasks are related because any disease found in task 2 would show positively in task 1. Thus, a Disease Head was proposed which utilizes this prior knowledge. Fig. A.6

<figure><img src="image_25.png" /><figcaption>Fig. A.6. The BNUAA designed module.</figcaption></figure>

shows the designed module, it is used as the classification head in CNN. The loss function for sub-challenge 1 is the cross-entropy (CE) loss. For sub-challenge 2 the authors used the BCE loss as the loss function. In addition, when using the proposed Disease Head, the loss function of sub-challenge 1 is modified to $\alpha L_{CE} + \beta L_{BCE}$. In the experiment, the ResNet (He et al., 2016) and ResNext (Xie et al., 2017) were used as the backbone and resized the image size to 512 × 512 pixels. The model was implemented in PyTorch and Python 3.6. The learning rate started from 0.001 and the networks were trained on one Tesla V100 with 40 epochs.

## Appendix A.4. Nekaravuru (Young-tack Oh et al.)

Oh et al. developed a two-branch network based on Efficient-Net (Tan and Le, 2019). Each branch was named LP for classifying large positive classes and SP for classifying small positive classes. The large positive class and small positive class were divided based on whether the number of positive samples exceeded 40 in the total data. Initially, for the given training

<figure><img src="image_26.png" /><figcaption>Fig. A.7. Pre-processing for three digital fundus camera images.</figcaption></figure>
images, data preprocessing was performed (Fig. A.7) and flip, rotation, shifting, blur, elastic transformations, grid distortion, and cutout data augmentation. The authors also set M_with that includes ‘Disease Risk’ class and M_without that excludes ‘Disease Risk’ class based on these divided classes. The final result is an ensemble of M_with and M_without (Fig. A.8).

<figure><img src="image_27.png" /><figcaption>Fig. A.8. LP is the classifier for large positive class and SP is the classifier for small positive class. M_with is composed of 13 class (a), and M_without is composed of 12 class (a*).</figcaption></figure>

Since negative labels dominate positive labels in most classes, the loss function should be adjusted to be more sensitive to detecting positive labels (Murphy, 2012). Thus, the authors used positive weighted binary cross entropy as a loss function given by:

$$L_c = \frac{1}{N} \sum_{n=0}^{N} [-\alpha_c \cdot y_{c,n} \cdot \log\sigma(x_{c,n}) - (1 - y_{c,n}) \cdot \log(1 - \sigma(x_{c,n}))] \quad (A.4)$$

Where c is the class number, N is the batch size, y_{c,n} and x_{c,n} represent the true label and predicted label for the c class for the n^th^ image. The positive weights α_c for class c is defined as follows:

$$\alpha_c = \frac{L_c^{neg}}{L_c^{pos}} \qquad (A.5)$$

Where L_c^{neg} and L_c^{pos} represent the number of negative labels and the number of positive labels for class c. However, if α_c is specified as above, the learning did not perform well because too much weight could be given to the little amount of data. So, authors set the threshold T_α and α_c is defined as follows:

$$\alpha_c = \begin{cases} 7 & \text{if } T_\alpha < \alpha_c \\ 3 & \text{if } 3 < \alpha_c < T_\alpha \\ 1 & \text{otherwise} \end{cases} \qquad (A.6)$$

In the large positive class, T_α was set to 10, and in the small positive class, T_α was set to 50. The final loss L_total is as follows:

$$L_{total} = \beta \cdot L_L + (1 - \beta)L_S \quad (A.7)$$

Where L_L and L_S represent the loss of LP classifier and the loss of SP classifier, respectively (Fig. A.8). β is a hyperparameter to indicate the loss ratio for each classifier. Experimentally, a large difference in the ratio gave worse results, and finally, β = 0.55 was set.

The method was implemented based on Pytorch and trained on four NVIDIA GeForce GTX 1080 GPUs. The Adam optimization algorithm and the initial learning rate is 3 × 10⁻⁴ was used. A Plateau scheduler was used that multiplies the learning rate by 0.7 whenever the learning area under the receiver operator curve (AUC) stagnates. EfficientNet pre-trained with ImageNet as a feature extractor was used.

Matsunaga et al. and Jin et al. have empirically found that combining predictions for multiple transformed versions of the test image helps improve performance (Matsunaga et al., 2017; Radosavovic et al., 2018; Wang et al., 2019). For this reason, the augmentation techniques that are applied during learning to the test image were able to get better results. In addition, the authors trained and ensembled a model with three different versions (EfficientNet-b0, b1, b2) of feature extractors to obtain a more generalized model.

## Appendix A.5. WWW (Hung Yeh et al.)

Yeh et al. first removed the redundant black edge from the fundus images. Then, rescaled images from the dataset to a size of 512 × 512 pixels as the input image. Then, CLAHE) and green channel of the image were used as pre-processing methods to improve the brightness and contrast. In addition, two functions of AutoAugment (Cubuk et al., 2019) to enhance the RFMiD dataset, called flip and rotation were used.

The authors used ImageNet pre-trained model and adopted EfficientNet (Tan and Le, 2019) and TResNet (Ridnik et al., 2021) to detect the ocular disease and classify the categorization of ocular disease. Moreover, different versions of EfficientNet were used for training, and the ensemble process was implemented to promote high performance. In the convolution layer of the deep learning network, the ReLU activation function was replaced with Mish (Misra, 2019) and used a ranger optimizer rather than SGD. Authors used an asymmetric loss function, as described by Ben-Baruch et al. (Ben-Baruch et al., 2020) to tackle the imbalanced data.

There are two sub-challenge in this challenge, namely, disease screening and disease classification. The authors used two kinds of image processing algorithms to provide different datasets and six versions of EfficientNet for experiments. For these two sub-challenge, the authors found that Efficient-Net B7 performed the best, which gets the best generalization and improved the prediction robustness of the deep learning network. In the disease classification task, the proposed algorithm achieved a better performance than the disease screening task. However, there are several categorizations in the disease screening task, which made it more difficult. After observing, the authors found that every single model had a specific ability in different tasks. Therefore, an ensemble algorithm was adopted
to merge the advantage of multiple diverse models to predict an outcome.

## Appendix A.6. IGSTfencing (Haoyu Wang et al.)

Wang et al. proposed a semi-supervised siamese graph convolutional network (SS-GCN) for both sub-challenges 1 and 2. SENet154 (Hu et al., 2018c) pre-trained on ImageNet dataset was used as the backbones for the proposed SS-GCN. The model adopted an MLCR-style (Niu et al., 2020) semi-supervised framework with two branches to learn useful information from both labeled and unlabeled data. Motivated by SSGRL (Chen et al., 2019), a semantic encoder was designed to extract feature vectors for each category based on both image feature maps and semantic information. Given a retinal image, backbone CNN was deployed together with semantic encoder to get semantic feature vectors of each branch. Then, feature vectors from two branches are fed into a GCN to model label relation. Finally, two outputs are fused to predict the final label distribution. The pipeline of the proposed SS-GCN framework is shown in Fig. A.9.

<figure><img src="image_28.png" /><figcaption>Fig. A.9. The pipeline of proposed semi-supervised siamese graph convolutional network (SS-GCN) framework. Both labeled and unlabeled images are encoded with the semantic encoder to obtain semantic features. Then, the multi-view GCN incorporates the features to model relations between each category to construct the final classifier. Finally, the feature vector after GAP is fed into the classifier to get the final results. $L_{mv}$, $L_{ce}$, and $L_{cr}$ are terms of the loss function to regularize different parts of SS-GCN.</figcaption></figure>

The BCE loss function was used for the backbone CNN and semantic encoder of each branch. In the semi-supervised framework, multi-view and co-regularization loss functions like MLCR for semi-supervised learning were adopted. The final loss function of the proposed SS-GCN framework can be formulated as:

$$L = \frac{1}{2} \sum_{i=0}^{1} L_{ce}^{i} + \lambda_{mv} L_{mv} + \lambda_{cr} L_{cr}, \quad (A.8)$$

The model was trained for 25 epochs using a SGD optimizer with an initial learning rate $\alpha_0^{(1)} = 0.07$ for multi-view GCN and $\alpha_1^{(1)} = 1.5 \times \alpha_0^{(1)}$ for semantic encoders to update the weights of the network. The learning rate was decayed as follows:

$$\alpha_i^{(n)} = 0.1^{\lfloor n/16 \rfloor} \times \alpha_i^{(1)}. \quad (A.9)$$

The learning rate is reduced by a factor of 0.1 at 16 epochs. Balancing factors $\lambda_{mv}$ and $\lambda_{cr}$ of loss function are set to 9.5. The authors utilize unlabeled data from ODIR-2019 (ODIR,

2019) which is publicly provided by Peking University. The batch size of each iteration is set to 16 (8 labeled images and 8 unlabeled images). All the experiments are conducted on an NVIDIA TESLA V100 GPU.

Before feeding the images into the network, the ROI was cropped from raw retinal images and resized into size (448×448 pixels). Then, three types of data augmentation were deployed to prevent the over-fitting issue. Firstly, a random horizontal flip was applied with a probability of 0.5 because it is ambiguous to distinguish whether the retinal image is from the left eye or the right eye. Secondly, random rotation was applied with a range of degrees in [-30, 30] because the shape of ROI is central symmetry. Thirdly, the network was trained by randomly cropping the image with a scale [0.5, 1.2] of (448 × 448 pixels) to enhance its generalization performance. Finally, the input image was ImageNet normalized. For inference of the evaluation set, a simple TTA strategy was used. First the images were resized to (512×512 pixels), then the center region with size (448×448 pixels) was cropped and then were ImageNet normalized. Then, flip the image horizontally, feed images of two directions into the network, and compute the mean value of outputs as the final result.

## Appendix A.7. MISIT (Dominik Müller et al.)

Müller et al. developed a multi-disease detection pipeline for retinal imaging which focuses on ensemble learning to combine the predictive capabilities of several heterogeneous deep CNN models. Their pipeline (Fig. A.10) consisted of three core pillars: The preprocessing methods, the neural network models, and the ensemble learning techniques.

<figure><img src="image_29.png" /><figcaption>Fig. A.10. Flowchart diagram of the implemented pipeline from Müller et al. for retinal multi-disease detection.</figcaption></figure>

For the augmentation, Müller et al. applied an up-sampling image augmentation approach (rotation, flipping, brightness, saturation, contrast, and hue) against the strong class imbalance bias, image padding to ensure aspect ratios and avoid distortion during the resizing, cropping to focus on the relevant retinal section, image resizing to 244 × 244, 299 × 299, or 380 × 380 pixels depending on the utilized neural network architecture and image intensity value normalization via Z-Score based on the average and standard deviation from ImageNet (Russakovsky et al., 2015). Next to the up-sampling image augmentation, images were real-time augmented during the training process to decrease the risk of overfitting and increase model generalizability.