RADR: A Robust Domain-Adversarial-based Framework for Automated Diabetic Retinopathy Severity Classification

Authors: sara mínguez monedero
fabian  westhaeusser
ehsan  yaghoubi
simone  frintrop
marina  zimmermann
m  monedero
Venue: MIDL_io_2024_Conference
Published: 2024-01-28 10:57:11

Abstract: Diabetic retinopathy (DR), a potentially vision-threatening condition, necessitates accurate diagnosis and staging, which deep-learning models can facilitate. However, these models often struggle with robustness in clinical practice due to distribution shifts caused by variations in data acquisition protocols and hardware. We propose RADR, a novel deep-learning framework for DR severity classification, aimed at generalization across diverse datasets and clinic cameras. Our work builds upon existing research: we combine several ideas to perform extensive dataset curation, preprocessing, and enrichment with camera information. We then use a domain adversarial training regime, which encourages our model to extract features that are both task-relevant and invariant to domain shifts. We explore our framework in its various levels of complexity in combination with multiple data augmentations policies in an ablative fashion. Experimental results demonstrate the effectiveness of our proposed method, achieving competitive performance to multiple state-of-the-art models on three unseen external datasets.

Main paper: 
 introduction: 
 Diabetic retinopathy (DR) is a medical condition that occurs due to microvascular retinal complications that are caused by diabetes mellitus. If the disease progresses, the result is irreversible vision loss, which is why early diagnosis of the disease is of utmost importance. DR is characterized by the presence of lesions in the eye: microaneurysms, hemorrhages, and soft and hard exudates. These are made visible on color fundus eye images and form the basis for evaluation of the severity of the disease by ophthalmologists (Wang and Lo, 2018; Lechner et al., 2017; Sun et al., 2022) . According to the International Clinical Diabetic Retinopathy (ICDR) scale, five levels of severity can be defined: no DR, mild, moderate, severe, and proliferative (Wilkinson et al., 2003) . Since the lesions are very small, manual diagnosis of DR is resource-intensive and time-consuming. This makes the development of algorithms to support the medical experts indispensable (Chen and Chang, 2022; Chetoui and Akhloufi, 2020) . Deep-learning (DL) models have achieved great success in various tasks in the field of medical image analysis, including DR grading (Li et al., 2021; Ragab et al., 2023; Litjens et al., 2017) . However, real-world clinical data includes many sources for variations, such as imaging standards, camera brands, and patient demographics, which lead to different distributions or covariate shifts, like e.g. in the color space (Guan and Liu, 2021) . As a consequence, DL models trained on one (or even several) specific domain(s) usually do not generalize well to other unseen domains, resulting in a lack of robustness for the application in real-world scenarios (Quinonero-Candela et al., 2008; Wang et al., 2022) . Domain adaptation techniques, which aim at transferring knowledge from the source domain to the target domain or making models domain-agnostic, have emerged as a promising solution to this problem studied for various imaging modalities (Aubreville et al., 2023; Wang and Deng, 2018) . Nonetheless, only limited research has been conducted in the field of domain adaptation for color fundus eye images (see Section 2). Therefore, the primary objective of this work is to develop a robust model for DR severity classification that generalizes to unseen domains. Our contributions include: DR severity classification framework: We introduce RADR, a novel Robust domain-Adversarial-based DL framework for DR severity classification, aimed to generalize across diverse datasets and fundus cameras. This builds upon existing research in the field, by, to the best of our knowledge, combining for the first time extensive dataset curation based on quality control labels provided by Fu et al. (2019) , camera domain information provided by Yang et al. (2020) , data preprocessing as well as domain adversarial training and data augmentation.Ablation study: We qualitatively and quantitatively assess the impact of various commonly used methods for domain generalization on our curated training dataset, namely domain adversarial learning, multi-camera (MC) training as well as AugMix and custom color augmentations, providing novel insights into their effectiveness in this field.Comparative analysis: We finally conduct a comparative analysis with multiple stateof-the-art models, demonstrating the effectiveness of our proposed method in achieving performance comparable to or surpassing existing approaches on three unseen external datasets. 
related work: 
 To the best of our knowledge, four distinct approaches for domain adaption in the field of DR grading have been published to date. First, Yang et al. (2020)  
radr: 
 In this work, we present RADR, a domain adversarial-based framework for automated diabetic retinopathy severity classification aiming for robust performance on unseen datasets. RADR is derived from the publicly available EyePACS dataset that we preprocess, curate for image quality and enrich with camera information. Using the camera labels as domains, we train RADR in an adversarial fashion to extract domain-agnostic features for its severity prediction. We finally compare our framework on three unseen commonly used DR datasets to the SOTA and evaluate our training regime in an ablative fashion. Figure 1 depicts the full pipeline. In the following, our approach is described in more detail.We train our model on the EyePACS dataset (Dugas et al., 2015) , a publicly available collection of 88,702 color fundus eye images. These images are classified into five classes, corresponding to the level of DR severity. We apply multiple steps of preprocessing and data curation to this dataset. Firstly, roughly 25% of the images in the EyePACS collection are considered to be ungradable due to the poor quality, artifacts, excessively bright or dark images, or out-of-focus images (Chetoui and Akhloufi, 2020) . Therefore, these poor quality images were eliminated according to the three-level quality labels provided by Fu et al. (2019) , removing the 'reject' category. Additionally, overly dark images were removed by thresholding on the average pixel value of images converted to grayscale. All images are cropped and resized into squares of size 512 × 512 pixels, centered around the retina, to remove noise and redundant image areas. Furthermore, the images of the EyePACS dataset were acquired using different camera brands. To utilize this inherent known heterogeneity in our model, we use the camera labels provided by Yang et al. (2020) to create a separate subdataset for each camera A, B, C, D and E. For aggregated RGB histograms per subdataset, refer to the Appendix. It should be noted that they stated differences between the provided labels and those employed in their original approach. After applying curation and preprocessing, in total 62,467 images remain of the EyePACS dataset. We split every camera subdataset by 70/15/15% into train, validation and test sets, stratified by the severity label to assure equal distribution. In addition to the five camera datasets (A-E), three of the most popular publicly available DR grading datasets are used as external datasets to evaluate the performance on unseen data and distributions. Those datasets are Messidor, Messidor 2 and APTOS (Abràmoff et al., 2013; Karthik, 2019) . This will also allow us to compare the results obtained in this paper with those obtained by the SOTA models presented by Atwany and Yaqub (2022) and Galappaththige et al. (2024) . In our pursuit to build a robust DR severity classification model, we first derive a singlecamera (SC) baseline model comprised of a ResNet-50-based feature extractor (He et al., 2016) and a fully connected DR classifier with 5 output nodes. The SC model is trained only on camera A (source) and tested on each of the remaining subdatasets individually. In addition, geometric transformations are applied when training the SC model to increase data variability. These correspond to rotation, vertical and horizontal flipping and cropping. In the next step, we perform multi-camera (MC) training on the subdatasets from camera A, B and C. This is expected to increase the data variability that the model sees during training and therefore to increase performance on the remaining unseen datasets from camera D and E. In the last step, inspired by Ganin et al. (2016) , we add a domain adversarial (DA) head to our network, composed of a Gradient Reversal Layer (GRL) and the domain classifier. This represents our final, most advanced model RADR. The domain classifier comprises two identical blocks, each with a linear layer, ReLu activation and dropout with probability of 0.5, followed by a final fully connected layer with three output nodes for domain classification. During backpropagation, the GRL inverts the sign of the gradient flowing from the domain head into the feature encoder (FE). Through parallel training of both classification heads, weights of the FE are adapted to produce domain-agnostic features that are still predictive for the main task of severity classification. Both the DR classifier and the domain classifier are optimized using the AdamW optimizer and Cross-Entropy loss. By utilizing both the severity labels and the domain information of all three training subdatasets in parallel during training, our framework diverges slightly from the original DANN implementation, which only used the labels of a single domain for training the main task. Additionally, a hyperparameter β was introduced to modulate the domain classification loss's impact. Therefore, the total loss L of the model can be calculated as L = L DRC + β • L DC where L DRC corresponds to the loss computed by the DR classifier and L DC to the loss computed by the domain classifier. β was empirically set to 0.3. Higher values tended towards degrading main task performance, while at lower values features could still be differentiated by the domain discriminator. We further wanted to analyze the influence of random data augmentations on the robustness of our models. For this, in addition to the default geometric transforms, we test the application of two different augmentation policies to the three presented levels of complexities of our pipeline, SC, MC, and DA training. The first policy employs color transformations, which randomly adjust the brightness, contrast, and saturation parameters of the images using ColorJitter of PyTorch's torchvision transforms (Paszke et al., 2019) . As our second policy, we evaluate AugMix from Hendrycks et al. (2019) , which is specifically designed to increase model robustness. In detail, AugMix creates multiple copies of an image, applies a unique data augmentation chain to each and then linearly combines them using random weights. For exact settings used in this work, refer to the Appendix. Finally, to evaluate the results obtained in classifying DR images, both quantitative and qualitative metrics are used. In this work, accuracy (ACC) and quadratic weighted kappa (QWK) are used as quantitative measurements, aligning with standard metrics in the literature to facilitate comparative analysis of the outcomes. We further utilize uniform manifold approximation and projection (UMAP) (McInnes et al., 2018) plots to visualize the latent representations before the final classification layer of all internal and external test data for qualitative analysis of the model. 
quantitative results: internal datasets: 
 Table 1 depicts the performance in terms of QWK of all training regimes and data augmentation policies on the EyePACS camera subdatasets. For the corresponding ACC results, please refer to the Appendix. All variations of the model were trained end-to-end using AdamW optimizer with hyperparameters tuned individually for best QWK on the validation splits of the EyePACS camera subdatasets. When comparing the three training regimes with only their default geometric augmentation and no added color augmentation, a consistent increase in performance on both seen and unseen datasets can be observed when using the MC approach over SC approach. Here, in contrast, camera D performance dropped by 18.2 percentage points. Overall, this indicates that the ColoJitter augmentations evaluated by us contribute negatively to overall performance and robustness. Finally, the application of the AugMix policy decreased average performance under the SC and DA training regime, though increasing average performance under the MC training regime by 1.8 percentage points, achieving comparable performance to our proposed main model RADR. A potential explanation for the negative influence of the augmentations under the DA training regime is that it opposes the domain discrimination task by blurring the differences between camera domains, limiting the potential benefit of the domain adversarial training. However, further research is required to verify this.We evaluate our top performing model RADR, which employs domain adversarial training without added color transformations, as well as our second best performing approach using multi-camera (MC) training with AugMix augmentations on three unseen public DR datasets, Messidor 1, Messidor 2 and APTOS (Table 2 ). Here, we also analyze classification accuracy besides QWK to enable a comparison to existing SOTA methods. Of our models, RADR achieves the highest average QWK and accuracy with 76.7% and 65.7%, respectively, though MC training with AugMix performs slightly better on the APTOS dataset. Notably, the highly similar scores of RADR on the QWK metric indicate consistent and robust performance across all unseen datasets. For benchmarking our method against SOTA, we compare against models from Atwany and Yaqub (2022) and Galappaththige et al. (2024) . We differentiate between approaches only using EyePACS as their single source (SS) training dataset and those utilizing a multi-source (MS) leave-one-out training regime by training on the remaining three datasets when predicting on one out of EyePACS, Messidor 1, Messidor 2 and APTOS. This concludes that our proposed framework is able to strongly compete with SOTA models, even by using less training data, or even surpass them under equal conditions.  
conclusion: 
 This paper presented RADR, a deep-learning framework for DR severity classification, which combines several ideas to perform extensive dataset curation, preprocessing, and enrichment with fundus camera information with a domain adversarial training regime. We explored our framework in its various levels of complexity in combination with multiple data augmentation policies in an ablative fashion, showing best performance when only using geometric transforms during training. Our model achieved competitive or higher performance to multiple SOTA models on three unseen external datasets, even when using less training data. We link this mostly to the reduction of noise in the dataset by the extensive preprocessing and filtering we conduct, which has, to the best of our knowledge, never been done before by similar approaches aiming for robustness in DR classification. We hypothesize that further improvements could be achieved by retraining our model on all five camera domains instead of only three, as well as by employing more advanced data augmentations, specifically aimed to function in unison with a domain adversarial training regime. 
 
 Appendix:  


Decision: The reviewers agree this paper is relevant, well written, and comprehensively and rigorously evaluated. The reviewers appreciated the efforts the authors made during the rebuttal phase, which improved the paper and addressed the reviewers’ concerns.

I follow the consensus recommendation of the reviewers and recommend acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The authors propose a domain adversarial framework for developing a robust diabetic retinopathy classification that is invariant to data shifts due to scanners and institutional differences. The paper provides a strong comparison of their results with relevant SOTA methods, popular data augmentation methods, and ablation studies.
main_review: main_review: The authors have appropriately addressed the feedback to the manuscript and made modifications where needed. Even though the ideal is not entirely novel, these results on data augmentations and discussions around it would be useful to other researchers for further analysis.
strength_weakness: strength_weakness: - In Figure 2 which depicts UMAP visualizations of latent space; each dataset also contains different DR severity samples which seem to be clustered very closely. One would ideally expect the same severity samples to be clustered together invariant of domain/dataset. However, this is not visible from these plots.
- Authors should also mention model parameters of their model in comparison to included SOTA as it has been shown to matter when analyzing data augmentations and domain generalization.
questions: questions: It would be useful for the research community if authors could provide comments on above mentioned weaknesses.
paper_summary: paper_summary: This paper adopts adversarial training to improve the accuracy of diabetic retinopathy severity stage grading. 
A main classification DL model processes fundus photographs and is coupled with a small classifier which is tasked to recognize the image domain (here camera labels). By using a reversal gradient layer the methods implements an adversarial form of training, which helps the main model in extracting features invariant to changes that may be specific to the camera. Combined with extensive data cleaning and augmentation, the approach results in enhanced model's robustness making it perform comparably to SOTA approaches on unseen publicly available datasets.
main_review: main_review: I would like to thank the authors for their response, I appreciate the effort which they put during such short rebuttal period of time. I was already supportive with presenting this paper but would like to upgrade my rating to strong accept.
strength_weakness: strength_weakness: Data augmentation plays a key role in this paper and in some cases data augmentation appeared to have degraded performance. 
I think the manuscript should provide more details about the data augmentation which could be a useful lesson on what worked and what did not work. Future researchers can start from there.
questions: questions: I would like the authors to focus a moment on the limitation of the approach and how it could be improved.
paper_summary: paper_summary: A novel Robust domain-Adversarial-based deep-learning framework for DR severity classification was built upon existing research. Qualitatively and quantitatively assessments of the impact of various commonly used methods for domain generalization on curated training dataset.
main_review: main_review: The authors have appropriately addressed the feedback to the manuscript and made modifications where needed, I appreciate the effort which they put in during such a short rebuttal period of time. The reviewer would like to recommend an acceptance based on the current manuscript.
strength_weakness: strength_weakness: 1. The hyperparameter $\beta$ hasn't been discussed in the experiment.
2. The authors claimed that the DRGen model utilized more training data and stem different versions of models, but failed to explain/prove this statement.
3. The variable $k$ hasn't been defined or cited in the paper, please explain and discuss.
4. Baselines were not introduced in either the introduction or related works.
questions: questions: Please see the weaknesses.

Comments: title: Rebuttal - Reviewer uwCk
Rebuttal - Reviewer m9oV
Rebuttal - Reviewer n9ci
Gentle reminder to post a rebuttal
comment: Thank you for your valuable insight and feedback, we appreciate your remarks very much. We hope the following answers and changes can help to mitigate the highlighted weaknesses:

About the DR severity labels in the UMAPs - Thank you for this remark. UMAP is an unsupervised clustering approach that clusters data based on variance present in the data. You are absolutely correct that in an ideal setting, where the strongest separator in the latent data is the DR severity label, the UMAP would cluster based on this label. However, since there is other bias in data, like bright/dark images etc., this might still lead to a stronger unsupervised clustering than the actual severity labels (Though the subsequent classification layer is still perfectly able to separate the latent space based on the label). Therefore we wouldn’t expect that in our model the UMAP clusters are mainly based on the DR severity. However, we wanted to prove that the domain information as the main source of variance has been reduced between the SD and the DA model, such that the UMAP doesn’t cluster based on the domain as its “easiest solution”, which was achieved in our opinion. This insightful question got us thinking for a while and we are more than happy to dive deeper into this with the reviewer in the following discussion period.

About the model parameters - Thank you for pointing out that we did not explain the hyperparameters and the number of trainable parameters in our model. We added a section to the Appendix covering all relevant hyperparameters for the models, the training and the augmentation policies. Also, the number of trainable parameters for the different parts of our model is included there. These refer to 23.5M for the ResNet50-based feature extractor, 10.2k for the DR severity classifier 3.1M for the domain classifier.
	
About dropout - You are correct that this is a very relevant mention, especially when talking about generalizability. We used dropout with a probability of 0.5. We added this number to the relevant section in the main text, as well as to the newly introduced list of used hyperparameters in the Appendix.

Changes in the manuscript:

We added the exact setting for β of 0.3 to the Methods. 

We added a clearer description about the leave-one-out training/evaluation procedure of the DRGen model to the Experiments section.

We added the names of the baselines architectures to the Related Work section.

We added a more detailed description of the used hyperparameters for augmentation and training in the Appendix.

We modified Section 3.2 to provide a clearer description of the adversarial approach.

We added the sentence to encourage further research on the influence of data augmentation at the end of the Internal Results.

We added aggregated RGB histograms for the camera datasets to the Appendix.
Thank you for your valuable insight and feedback, we appreciate your remarks very much. We hope the following answers and changes can help to mitigate the highlighted weaknesses:

About the data augmentation - Thank you for pointing this out, this is a very relevant question. We did quality check the images before to verify visually that the images did not deviate into “unrealistic” regimes. But given research on other datasets like ImageNet, where very strong color augmentations, even to the point of “unrealistic” looking images provided good generalization results [Cubuk et al. 2019 - “AutoAugment: Learning Augmentation Strategies from Data”], it is difficult to quantify visually beforehand whether an augmentation will have the desired effect. We added the exact hyperparameter settings for AugMix and ColorJitter, based on your remark, to the Appendix section and also added a remark pointing to the Appendix in the main text. To understand why augmentation did not provide the desired effects in some cases or even had detrimental influence is a very interesting topic that would, however, require further analysis. We added the sentence “However, further research is required to verify this.” at the end of the Internal Results section to encourage this. If we would need to formulate a suspicion for the detrimental effects, besides the one given in the manuscript (Augmentation opposes DA), it would be that the chosen augmentation ranges did not properly cover the domains of the unseen cameras and rather “confused” the network by introducing noise in the wrong “direction”.

About the correlation of camera datasets - This is a very interesting question, thank you for that! Measuring “correlation” between datasets or domains is a key aspect in the field of domain generalization or robustness and not a straightforward question to answer in our opinion. We think that correlation could be measured on different factors, such as average color intensities per dataset, texture of the images, systematic geometric differences and so on. The most promising individual factor would probably be color distribution. For this, we calculated the aggregated RGB histograms for all camera datasets and attached the plot to the Appendix. Here, shifts in the red, blue and green channel between the camera datasets are visible.

About the limitations of the approach and how it could be improved - Thank you for this question. We think that the most significant limitation of the proposed approach is that it requires data from multiple domains that express a sufficient domain shift which can be utilized by the adversarial training. Additionally, the domains need to be known and labeled in advance. If only a single domain is available for training, the method can not be applied. On the side of potential improvements it would be desirable to extract domains directly from the underlying data, similar to what the authors in (Marini et al. - “H&E-Adversarial Network: A Convolutional Neural Network To Learn Stain-Invariant Features Through Hematoxylin & Eosin Regression”) propose for histopathological images. Here, instead of defining the domains by acquisition protocol, they calculate the stain matrix for every image, which incorporates color information. They then perform adversarial regression on the stain matrix, such that basically every image provides its own “domain” that should be unlearned. A similar approach would be thinkable for fundus images, by performing adversarial regression on e.g. the hue channel of the images.

Changes in the manuscript:

We added the exact setting for β of 0.3 to the Methods. 

We added a clearer description about the leave-one-out training/evaluation procedure of the DRGen model to the Experiments section.

We added the names of the baselines architectures to the Related Work section.

We added a more detailed description of the used hyperparameters for augmentation and training in the Appendix.

We modified Section 3.2 to provide a clearer description of the adversarial approach.

We added the sentence to encourage further research on the influence of data augmentation at the end of the Internal Results.

We added aggregated RGB histograms for the camera datasets to the Appendix.
Thank you for your valuable insight and feedback, we appreciate your remarks very much. We hope the following answers and changes can help to mitigate the highlighted weaknesses:

About β - Thank you for pointing this out! We explored different values for β between 0.1 and 1. We empirically determined 0.3 as the best value for β in our use case. At higher values, the influence of the DA training became too strong, such that features also became uninformative for the main task. At lower values, the features still contained too much domain information, such that the domain discriminator could easily differentiate them. We added the following to the manuscript: “β was set empirically to 0.3. Higher values tended towards degrading main task performance, while at lower values features could still be differentiated by the domain discriminator.“ We additionally added a more detailed description of the used hyperparameters for augmentation and training to the Appendix.

About the utilized training data - The authors of the DRGen model [Atwany et al. 2022 - “DRGen: Domain Generalization in Diabetic Retinopathy Classification”] evaluated their model’s performance under a leave-one-out regime on the four datasets Messidor, Messidor 2, APTOS and EyePACS. That means that their reported performance per test dataset always stems from training on the three remaining datasets (refer to Table 3 & 4 in their paper). Since the full EyePACs dataset is always in their training data for the reported external datasets Messidor, Messidor 2, APTOS in our paper, and our method only utilizes three out of five camera domains from EyePACS for training, the utilize significantly more training data than us. Moreover, since they retrain their model for every test dataset on the three other datasets, every reported accuracy stems from a different version of their model. However, our reported results on Messidor, Messidor 2, APTOS are all created by inference on the same RADR model. We changed the sentence in the manuscript to highlight this to: “By employing a leave-one-out training and evaluation regime on the four datasets EyePACS, Messidor 1&2 and APTOS, they not only utilized significantly more training data than us, the reported accuracies per unseen dataset also stem from different versions of their model, while our results are all from the same version.”

About the variable 𝜅 - Thank you for mentioning that we should explain or introduce this variable. It was referring to the number of image copies that are created and augmented during the AugMix procedure, before being merged into a single augmented image. In our work, we used 3 copies, since this was the default setting of the official Pytorch implementation. We note that we did not tune this parameter due to time constraints and focus on other aspects of this work that we deemed more important. To avoid confusion, we removed the “k” from the main text. As mentioned, we now added all augmentation, training and model based hyperparameters to the Appendix.

About the introduction of baselines - Thank you for this comment. Based on this, we updated the Related Work section to clearly introduce the baselines with their proposed model names (DRGen of Atwany and Yaqub (2022) and SPSD-ViT Galappaththige et al. (2024)) for easier referencing.

About description of the adversarial training - Thank you for pointing this out. We modified Section 3.2 to describe the used adversarial approach clearer: “In the last step, inspired by Ganin et al. (2016), we add a domain adversarial (DA) head to our network, composed of a Gradient Reversal Layer (GRL) and the domain classifier. This represents our final, most advanced model RADR. The domain classifier comprises two identical blocks, each with a linear layer, ReLu activation, and dropout, followed by a final fully connected layer with three output nodes for domain classification. During backpropagation, the GRL inverts the sign of the gradient flowing from the domain head into the feature encoder (FE). Through parallel training of both classification heads, weights of the FE are adapted to produce domain-agnostic features that are still predictive for the main task of severity classification.”

Changes in the manuscript:

We added the exact setting for β of 0.3 to the Methods. 

We added a clearer description about the leave-one-out training/evaluation procedure of the DRGen model to the Experiments section.

We added the names of the baselines architectures to the Related Work section.

We added a more detailed description of the used hyperparameters for augmentation and training in the Appendix.

We modified Section 3.2 to provide a clearer description of the adversarial approach.

We added the sentence to encourage further research on the influence of data augmentation at the end of the Internal Results.

We added aggregated RGB histograms for the camera datasets to the Appendix.
Dear authors,

This is just a gentle reminder that the rebuttal deadline is coming up. I encourage you to take this opportunity so you can engage with the reviewers in the upcoming discussion phase.

Best wishes,
AC