# A Multi-Task Learning and Multi-Branch Network for DR and DME Joint Grading
## 3 Proposed Method
### Feature Fusion Module
#### 3.2.1 Self-Feature Extraction Network  
DR and DME have their own specific characteristics. DR grading is based on the presence of exudate, hemorrhage, microaneurysms, and other lesions.DME is graded according to the location relationship between the central macular and the hard exudate. However, the features \(F_{DR}\) and \(F_{DME}\) extracted by the MbN are only the underlying representation of the fundus images, and the detailed self-features of each disease are not easy to capture. To better learn the representative characteristics of each disease, in this paper, the SFEN module is designed to obtain specific discriminatory characteristics of DME and DR. Figure 4 shows the specific architecture of the SFEN module, which is composed of the \(CA\) (channel attention) and \(SA\) (spatial attention) mechanisms. SFEN takes the feature map \(F\) as the input and uses \(CA\) and \(SA\) to enhance the inter-channel and interspatial relationship of the features related to each disease. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Branch 1** & **Branch 2** & **Branch 3** & **Branch 4** \\ \hline Input (224 \(\times\) 224 \(\times\) 3) & Input (branch 2–output) \\ \hline conv3 – 64 \(\times\) 2 & conv3 – 64 \(\times\) 2 & conv3 – 512 \(\times\) 3 & conv3 – 512 \(\times\) 3 \\ \hline \multicolumn{3}{c}{Maxpool (pool\_size = (2, 2), strides = 2)} \\ \hline conv3 – 128 \(\times\) 2 & conv3 – 128 \(\times\) 2 & conv3 – 512 \(\times\) 3 & conv3 – 512 \(\times\) 3 \\ \hline Maxpool (pool\_size = (2, 2), strides = 2) & Maxpool (pool\_size = (3, 3), strides = 1) \\ \hline conv3 – 256 \(\times\) 3 & conv3 – 256 \(\times\) 3 & \\ \hline Maxpool (pool\_size = (2, 2), strides = 2) & UpSampling \\ \hline conv3 – 512 \(\times\) 3 & & \\ \hline Output & Output & Output \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Network structure settings. Figure 4: SFEN Module. It consists of two parts, the channel attention and spatial attention mechanisms. The \(CA\) operation is mainly divided into 3 steps:  
1. F is processed by the global averaging pooling to obtain the feature map with global sensory field \(Z\). The calculation is described in Equation (1). \[Z=\text{f}_{\text{GA}}(\text{F})=\frac{1}{\text{H}\times\text{W}}\sum_{i=1}^{ \text{H}}\sum_{j=1}^{\text{W}}\text{F}_{\text{ij}}\] (1) where, \(Z\) denotes the global feature, \(f_{GA}\) denotes the global averaging pooling operation, \(F\) denotes the input feature map, \(H\) is the height of \(F\), \(W\) is the width of \(F\), and \(\text{F}_{\text{ij}}\) is the feature vector of the pixel in the _i-th_ row and _j-th_ column. 2. \(Z\) is processed by the use of two \(FC\) operation to get the dependencies between individual channels. The process can be given by Equation (2). \[CA=\sigma_{1}(W_{2}\sigma_{2}(W_{1}Z))\] (2) here, \(W_{1}\in R^{1\times 1\times\hat{\tau}}\) is used for dimensionality reduction and \(W_{2}\in R^{1\times 1\times\hat{\tau}}\) is used for dimensionality increase. \(r\) denotes the scaling factor (set to 4 here), \(\sigma_{1}\) is the sigmoid function, and \(\sigma_{2}\) is the ReLU activation function. 3. The corresponding weight \(CA\) is multiplied with the original input feature \(F\) to obtain the channeled disease grading discriminant feature \(F^{\prime}\). The calculation is described in Equation (3). \[F^{\prime}=CA\otimes F\] (3) The \(SA\)[36; 37] is used to focus on the more important location of the disease feature, which is complementary to the \(CA\). The structure of \(SA\) is shown in the blue area of Figure 4. The \(SA\) operation is mainly divided into 3 steps: (4) The input features \(F^{\prime}\) are convolved by two convolution branches to obtain two spatial feature maps \(C_{1}\) and \(C_{2}\). Every convolution branch has two convolutional layers, one with a kernel of \(1\times k\) and the other with \(k\times 1\). This convolutional layer design can enlarge the acceptance domain without increasing the training parameters. The corresponding calculation can be given by Equations (4) and (5).