# A Multi-Task Learning and Multi-Branch Network for DR and DME Joint Grading
Figure 2: Example images of the degree of DR and DME pathology, (**a**,**b**) labeled DR2 DME2, (**c**,**d**) labeled DR4 DME0. (**a**) DR2 DME2; (**b**) DR2 DME2; (**c**) DR4 DME0; (**d**) DR4 DME0. lesions. [PERSON] et al. [7] used the pre-trained InceptionV3 model on the ImageNet [8] dataset to perform DR classification. [PERSON] et al. [9] built a high-quality dataset and performed two and four classifications for DR by using an integrated model. [PERSON] et al. [10] proposed an integrated algorithm based on multiple improved Inception-v4 for DR lesion detection on retinal fundus images. [PERSON] et al. [11] proposed a CNN-based method to simultaneously diagnose DR lesions and highlight suspicious areas. [PERSON] et al. [12] designed an attention fusion-based network with better noise immunity for DR grading, which fused the lesion features by CNN and color fundus images for 5-level DR grading. [PERSON] et al. [13] used a semi-supervised learning method to improve the performance of DR grading and lesion segmentation through collaborative learning. For DME grading, [PERSON] et al. [14] used a two-stage approach to classify DME. In the first stage, an eight-layer CNN model was trained on the e-ophtha database to detect the hard exfiltration, which used 48 \(\times\) 48 pixel RGB slices as input; in the second stage, the pre-trained CNN model was used as predictor to generate a new dataset of 1200 grayscale mask images; in the final stage, the DME detection model based on the AlexNet architecture was trained, which consists of a 17-layer CNN using a 512 \(\times\) 512 pixel RGB fundus image plus the previously generated grayscale mask as the 4 th input channel for training. [PERSON] et al. [15] constructed a deep residual network through cascading to identify DME. In the model, a fully convolved residual network was used to perform hard exudate mask segmentation, and then according to the segmentation results the region with the highest pixel-centered probability was cropped and fed it into another deep residual network for grading. [PERSON] et al. [16] proposed a multiscale collaborative grading model. A multiscale feature extraction model was designed to extract different features, including hard exudation masks, macular masks, and macular images. XGBoost classifier was used to perform the classification training according to the features and the original images. There has been a lot of research on DR and DME grading and some progress has been made. However, most works are concern at the separate grading of DR or DME. In fact, physicians perform the diagnosis of DR and DME at the same time. Therefore, it is necessary to design a computer vision-based method to automatically grade DR and DME simultaneously. In this paper, a multi-task learning and multi-branch network (MaMNet) is proposed to achieve simultaneous grading of DR and DME. The contribution of this paper are as follows:  
1. Build a multi-task learning and multi-branch network (MaMNet) for simultaneous grading of DME and DR. By tapping the relationship between the two grading tasks of DME and DR, multi-task learning can make models more robust and improve the grading accuracy. 2. Design a four-architecture multi-branch network to increase the expression of the underlying features of DME and DR.
3. Design a feature fusion module to fuse the self-features, the cross-feature, and the global features involved in DR or DME to enhance the joint grading accuracy. The rest sections are organized as follows. Section 2 focuses on the application of multi-task learning, multi-branch networks, and attention mechanisms in medical image processing. Section 3 provides a detailed description of MaMNet. Section 4 illustrates the relevant experimental results. Section 5 concludes this paper.