<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Interpretation of Feature Contribution
## Towards Diagnosis of Diabetic Retinopathy from Exudates in Retinal Images

Kanupriya Mittal and V. Mary Anita Rajam

Department of CSE, CEG, Anna University, Chennai, India
kanu.05@gmail.com

**Abstract.** Diabetic retinopathy is a complication related to diabetes and is one of the leading causes of blindness worldwide. The presence of exudates is one of the earliest clinical signs of diabetic retinopathy. Machine learning algorithms have been used to classify the presence or absence of exudates in retinal images. The black box nature of the image classification based on machine learning models limits the clinical applicability. This paper proposes an explainable machine learning model to detect and classify exudates. Dynamic fuzzy histogram equalization and morphological operations are used for the segmentation of exudates. The XGBoost classifier with a feature vector of eighteen features is used to classify the presence or absence of exudates in images. A SHAP tree explainer is used for the interpretation of the classifier's output. The SHAP tree explainer helps in the identification and ranking of the features extracted from the images towards output probabilities.

**Keywords:** Model interpretability · Shapley values · Retinal image · Fuzzy histogram equalization · Exudates · Classification

# 1 Introduction

Machine learning (ML) models have been widely used in computer aided medical diagnosis systems. Despite that, doctors are doubtful to adhere to the computer aided medical diagnosis systems as the process behind the learning made by computer models is not transparent. Computer aided medical diagnosis systems should be interpretable, understandable and transparent to overcome the black box nature of such systems. Explainability of the machine learning models will help to enhance the trust of clinical experts in the machine learning system.

Interpretation of ML models can be in model-specific manner or model-agnostic manner. Model-specific methods are restricted to specific models and the explanations are derived from the internal model parameters [5]. Model-agnostic methods are applicable to any machine learning model [22]. Some of the model-agnostic models are partial dependence plots (PDP), local interpretable
model-agnostic explanations (LIME) [26], model understanding through sub-space explanations (MUSE) [16] and Shapley additive explanations (SHAP) [17].

In this work, we have proposed an interpretable machine learning model for exudates detection from retinal images. Exudates are white or yellowish spots with high contrast, and have varying shapes and sizes. These are the lipid deposits formed due to the damaged blood vessels in the retina. The presence of exudates is one of the indicators of diabetic retinopathy (DR), which is one of the leading causes of blindness in the world. The size of the exudates increases with the disease severity level. The automated process of exudates detection would help in early treatment of DR.

The proposed method uses a dynamic fuzzy histogram equalization technique during the pre-processing step, mathematical morphological operations for exudate candidate detection and then exudate segmentation, extreme gradient boosting (XGBoost) classifier for classification and then a SHAP tree explainer for model interpretation. The method is tested on images from publicly available datasets. The technique is fast and requires very low computing power.

The paper is organised as follows: The related work is briefed in Sect. 2. Section 3 gives details about the proposed methodology. Section 4 provides the details on the datasets used and evaluation metrics. Section 5 provides the details on the results and discussion. Section 6 discusses the conclusion and future scope.

## 2 Related Work

Medical image classification of retinal exudates has been around as a research area but only limited work is available in the literature on the interpretable machine learning model for retinal exudates detection and other medical areas. Jiang [13] proposed class activation mapping based interpretable deep learning model for diabetic retinopathy classification. Sayres [27] used integrated gradient based interpretable model for understanding diabetic retinopathy grading using deep learning approach.

Several methods have been proposed by various researchers for screening exudates in colour retinal images. A number of methods (thresholding, region growing based, morphology, machine learning) have been proposed for exudates detection. Mathematical morphology-based approaches have been applied by a large number of researchers for exudates segmentation [21].

Welfer [30] proposed a three-step process for the segmentation of exudates on LUV colour space using mathematical morphology technique. The shortcoming of the method was that it had high false positive cases. Wisaeng [32] used a morphology mean shift algorithm for exudates detection. Coarse segmentation was first achieved using mean shift algorithm and then mathematical morphology was applied to achieve fine segmentation. The limitation of the method was that it did not work well in the case of small exudates.

Sopharak [29] proposed fuzzy c-means and support vector machines (SVM) based techniques. Image was enhanced using the local contrast method and then FCM was used. SVM was applied on the segmented exudates for better results.
The limitation of this method was that its success depended on the accurate removal of the optic disc (OD) and the blood vessels. Wisaeng [31] used fuzzy c-means clustering with mathematical morphology for exudate detection. Sidibe [28] performed bright lesions detection from retinal images. Sparse coding techniques were used for classification. Kusakunniran [15] approach used multi-layer perceptron (MLP) to identify initial seeds. Then the iterative graph cut approach was used for segmentation. The technique used both supervised and unsupervised learning for exudates segmentation. Javidi [11] proposed approach based on morphological component analysis (MCA) with the adaptive representations obtained from dictionary learning. Marin [20] used feature extraction with supervised classification technique for exudates detection.

Deep learning methods are recently applied for exudates detection. Prentavsic [25] have used deep convolutional neural networks incorporated with high level anatomical knowledge for exudate detection. Feng's [6] approach was based on fully convolutional neural network.

The state-of-the-art methods discussed above have some limitations and challenges. The requirement of good quality images is one of the limitations. Some of the methods work with bright lesions only and there are false positive cases detected due to dull boundaries and colour. The drawback of machine learning methods is their black box nature, and also they require high computing power and time for processing. Also, the supervised learning approaches are highly dependent on training data for classification. Although the deep learning methods show promising results, the state-of-the-art conventional image analysis methods have reported high sensitivity and accuracy than the deep learning methods. The main challenges for deep learning methods are the limited availability of the labelled training data, the imbalanced representations of different classes and their inability to explain the results to end-user.

Keeping the above challenges and limitations in mind, we propose an interpretable exudates detection technique, which is self explainable for the decision it makes, will smoothen the retinal fundus examination process and assist ophthalmologists.

## 3 Proposed Methodology (SHAP-DFHE-MR Model)

The proposed system is based on dynamic fuzzy histogram equalization technique (DFHE), mathematical morphology and Shapley value based tree explainer and hence is named as SHAP-DFHE-MR model. The schema of the SHAP-DFHE-MR model is presented in Fig. 1. The proposed methodology consists of: preprocessing, optic disc segmentation, exudate candidate detection, feature extraction, classification and SHAP tree explainer interpretation. The SHAP-DFHE-MR model computes the contribution of each feature extracted from the images towards the classification of the presence or the absence of exudates.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Proposed Methodology (SHAP-DFHE-MR)</figcaption></figure>

## 3.1 Preprocessing

The input images belong to different datasets and are taken with different camera types. The intensity of the images can vary. Hence, preprocessing of the acquired retinal images is important.

The original image ($I_{orig}$) is first transformed to the hue, saturation, intensity (HSI) colour space and then the intensity band is selected. HSI colour space is selected as the exudates are bright lesions with high contrast from the background. Median filter removes the noise from the I-band image. Then the dynamic fuzzy histogram equalization (DFHE) technique is applied to enhance the image to get the preprocessed image (Fig. 2(b)). The preprocessed image ($I_{prepro}$) is then used as input for the next stage.

**Dynamic Fuzzy Histogram Equalization (DFHE) Technique.** The dynamic fuzzy histogram equalization (DFHE) is a methodology for enhancing the image contrast and preserving the image brightness [10, 19]. It is a three-step process. In the first step, a fuzzy histogram computation is performed with a triangular fuzzy membership function to handle the inexactness of grey level values. In the second phase, the fuzzy histogram is divided into sub-histograms and dynamic histogram equalization is done for each sub-histogram. Finally, normalization of image brightness is performed.

Assume an image *I* with grey levels in the range [0, L-1]. The concept of fuzzy sets is applied to deal with the imprecise grey values [12]. The classical concept of histogram is extended to a fuzzy setting by considering grey values as fuzzy numbers. So let a grey value *I*(*i*, *j*) be considered as a fuzzy number $\tilde{I}(*i*, *j*)$. A
fuzzy histogram is a sequence of real numbers $k(i)$ where $i \in \{0, 1, \dots, L-1\}$ and $k(i)$ is the frequency of occurrence of grey levels that are around $i$. Let $\mu_{\tilde{I}(i,j)}$ be a triangular fuzzy membership function then the fuzzy histogram is given as in Algorithm 1.

The first and second derivatives of the fuzzy histogram are first calculated. Then the sub histograms are obtained using the local maxima of the fuzzy histogram. The equalization of these sub histograms thus obtained is performed by the dynamic histogram equalization technique given by Ibrahim et al. [10]. Normalization is then applied on the equalized image obtained so that the mean brightness is the same as the original input image. This method gives better results as we are equalizing only the intensity band I, whereas, in other methods R, G, B planes are equalized separately.

## 3.2 Optic Disc Segmentation

The segmentation and elimination of the optic disc (OD) is an important step in exudates detection as it has similar features (brightness and contrast) as that of exudates, and hence, can lead to false positive cases. OD is the brightest and largest structure in the retinal image. The mathematical morphological closing operation is applied on the pre-processed image ($I_{prepro}$) with the flat disk structuring element of size 4. Then thresholding give the optic disc region and a masked image is formed by complementing the threshold image. Then, mathematical morphological dilation operation with the flat disk structuring element of size 4 is applied to expand the optic disc region and dilation is done again with the flat disk structuring element of size 8 to achieve better results and the reconstructed image is achieved. Then the difference is taken between the original pre-processed image ($I_{prepro}$) and the reconstructed image and Otsu threshold is applied. The Otsu algorithm is used to automatically select the image threshold value [23]. The candidate optic disc regions thus obtained are shown in Fig. 2(c).

In our work, we have used area as a shape feature to find the optic disc, the region with the largest area. All the connected components of the image in Fig. 2(c) are found using 8-pixel connectivity. A connected component is a cluster of pixels connected to each other through 8-pixel connectivity. The 8-pixel connectivity groups the pixels if they are connected along their edges or corners. For each of the connected components, area is calculated. Then the region with the largest area is selected as the OD. The image obtained is then inverted and binarized to show the localised optic disc. The image ($I_{od}$) with the optic disc segmented is shown in Fig. 2(d).

## 3.3 Exudates Detection

The pre-processed image ($I_{prepro}$) along with the OD segmented image ($I_{od}$) is processed for the extraction of exudates. The high-contrast retinal blood vessels are first removed from the pre-processed image with a morphological closing operator and a disk-shaped structuring element of size 8, as in Fig. 3(a).
<table><caption>**Algorithm 1: Pre-processing Algorithm**</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input</td><td>Original Image (I<sub>orig</sub>)</td></tr><tr><td>Output</td><td>Pre-processed Image (I<sub>prepro</sub>)</td></tr><tr><td>1</td><td>foreach (I<sub>orig</sub>) do</td></tr><tr><td>2</td><td>/* convert RGB to HSI colour space */</td></tr><tr><td>3</td><td>HSIImg = RGB2HSI(I<sub>orig</sub>)</td></tr><tr><td>4</td><td>FImg = MedianFilter(HSIImg(:,3))</td></tr><tr><td>5</td><td>/* DFHE technique */</td></tr><tr><td>6</td><td>N ← total number of pixels in FImg</td></tr><tr><td>7</td><td>[0, L − 1] ← grey level range in FImg</td></tr><tr><td>8</td><td>foreach iε[0, 1, ..., L − 1] do</td></tr><tr><td>9</td><td>k(i) ← frequency of occurrence of grey levels</td></tr><tr><td>10</td><td>F(i) ← fuzzy histogram</td></tr><tr><td>11</td><td>[I<sub>min</sub>, I<sub>max</sub>] ← fuzzy histogram intensity range</td></tr><tr><td>12</td><td>μ<sub>Ĩ(i,j)</sub> ← triangular fuzzy membership function</td></tr><tr><td>13</td><td>/* Fuzzy histogram creation */</td></tr><tr><td>14</td><td>μ<sub>Ĩ(i,j)</sub> = max(0, 1 − |I(i, j) − i|/4)</td></tr><tr><td>15</td><td>F(i) ← k(i) + Σ<sub>i</sub> Σ<sub>j</sub> μ<sub>Ĩ(i,j)</sub></td></tr><tr><td>16</td><td>/* Histogram partitioning */</td></tr><tr><td>17</td><td>F'(i) = dF(i)/di</td></tr><tr><td>18</td><td>F''(i) = d<sup>2</sup>F(i)/di<sup>2</sup></td></tr><tr><td>19</td><td>local maxima points [m<sub>0</sub>, .., m<sub>n</sub>] ∀F'(i) = 0, F''(i) &lt; 0</td></tr><tr><td>20</td><td>sub histograms = [I<sub>min</sub>, m<sub>0</sub>],..., [m<sub>n+1</sub>, I<sub>max</sub>]</td></tr><tr><td>21</td><td>Map sub histograms</td></tr><tr><td>22</td><td>I<sub>equalized</sub> ← equalized image</td></tr><tr><td>23</td><td>end</td></tr><tr><td>24</td><td>/* Normalization */</td></tr><tr><td>25</td><td>m1 ← mean brightness of I<sub>orig</sub></td></tr><tr><td>26</td><td>m2 ← mean brightness of I<sub>equalized</sub></td></tr><tr><td>27</td><td>I<sub>prepro</sub> = m1/m2 I<sub>equalized</sub></td></tr><tr><td>28</td><td>end</td></tr></tbody></table>

In this work, we have used the local variance to get the mean standard deviation. This helps to get the local image characteristics of the exudates clusters as in Fig. 3(b). A small sliding window is used to compute the local variance of an image. The window is moved throughout the entire image by moving one pixel at a time. Then the mean of all the local variances computed is taken to get the local variance of the image as in algorithm 2 [1,33].

The local variant image is then thresholded using Otsu algorithm and then flood fill operation is applied to get the candidate exudates regions. Thresholding will remove all the regions with low local variation.

The optic disc detected in the previous step ($I_{od}$) is dilated with a disk-shaped structuring element of size 15. This image is then subtracted from the image ($I_{fill}$) as obtained above to remove the optic disc, as per Eq. 1. The image ($I_{candex}$) thus obtained has all the candidate exudates regions and is shown in Fig. 3(c). The resultant image is then used for final level exudates detection.
<figure><img src="image_3.png" /><figcaption>Fig. 2. Optic disc extraction stage (a)original Image ($I_{orig}$) (b)DFHE enhanced (pre-processed) image (c)candidate OD regions (d)OD segmented image ($I_{od}$)</figcaption></figure>

$$I_{candex} = I_{flfill} - dilate(I_{od}) \qquad (1)$$

Morphological reconstruction is then applied on the image obtained. The morphological reconstruction is based on the dilation process and it has some unique features. The technique uses connectivity concept with two images (a marker and a mask). Basically, repeated dilation of marker image is done till its contour fits the mask image. Hence, final dilation results in reconstructed image. The image $I_{candex}$ is used to create a marker image, $I_{marker}$ as per Eq. 2.

$$I_{marker} = I_{prepro} \cdot (1 - I_{candex}) \qquad (2)$$

This image, $I_{marker}$ is now morphologically reconstructed upon the preprocessed image, $I_{prepro}$. The resultant image, $I_{reconstruct}$ is as shown in Fig. 3(d). The difference is taken between the images $I_{prepro}$ and $I_{reconstruct}$ and then a threshold is applied on this difference image to get the final exudate detected image, $I_{ex}$, as in Fig. 3(e). Then, all the exudates pixels in $I_{ex}$ are surfaced on the original image ($I_{orig}$) as in Fig. 3(f).

## 3.4 Feature Extraction and Classification

Features are unique attributes of any image and the feature extraction process aims at pixel depiction using a feature vector. In this work, the automated exudate detection system requires features of healthy and diseased images to
<table><caption>**Algorithm 2: Exudate Detection**</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input</td><td>Pre-processed image (I<sub>prepro</sub>)</td></tr><tr><td>Input</td><td>OD segmented image (I<sub>od</sub>)</td></tr><tr><td>Output</td><td>Ex detected image (I<sub>ex</sub>)</td></tr><tr><td>1</td><td>foreach I<sub>prepro</sub>, I<sub>od</sub> do</td></tr><tr><td>2</td><td>&nbsp;&nbsp;&nbsp;&nbsp;se2 &larr; structuring element size 8</td></tr><tr><td>3</td><td>&nbsp;&nbsp;&nbsp;&nbsp;i<sub>nobv</sub> = (I<sub>prepro</sub> &oplus; se2) &ominus; se2</td></tr><tr><td>4</td><td>&nbsp;&nbsp;&nbsp;&nbsp;/* Local variance */</td></tr><tr><td>5</td><td>&nbsp;&nbsp;&nbsp;&nbsp;foreach J &isin; i<sub>nobv</sub> do</td></tr><tr><td>6</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;J &larr; block of (M,N) in i<sub>nobv</sub></td></tr><tr><td>7</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&mu; &larr; mean of pixel intensities of J</td></tr><tr><td>8</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;J<sub>m,n</sub> &larr; pixel intensity at (m,n)</td></tr><tr><td>9</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&mu; = <sup>1</sup>&frasl;<sub>MN</sub> &sum;<sub>m=0</sub><sup>M-1</sup> &sum;<sub>n=0</sub><sup>N-1</sup> J<sub>m,n</sub></td></tr><tr><td>10</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&sigma;<sup>2</sup> = <sup>1</sup>&frasl;<sub>MN-1</sub> &sum;<sub>m=0</sub><sup>M-1</sup> &sum;<sub>n=0</sub><sup>N-1</sup> (J<sub>m,n</sub> &minus; &mu;)<sup>2</sup></td></tr><tr><td>11</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;i<sub>localvar</sub> &larr; mean of all &sigma;<sup>2</sup></td></tr><tr><td>12</td><td>&nbsp;&nbsp;&nbsp;&nbsp;end</td></tr><tr><td>13</td><td>i<sub>otsuth</sub> &larr; Otsu threholded on i<sub>localvar</sub></td></tr><tr><td>14</td><td>I<sub>flfill</sub> &larr; flood-fill on i<sub>otsuth</sub></td></tr><tr><td>15</td><td>se3 &larr; structuring element size 15</td></tr><tr><td>16</td><td>i<sub>dilate</sub> = (I<sub>od</sub> &oplus; se3)</td></tr><tr><td>17</td><td>I<sub>candex</sub> = (I<sub>flfill</sub> &minus; i<sub>dilate</sub>) /*Remove optic disc */</td></tr><tr><td>18</td><td>I<sub>marker</sub> &larr; marker image of I<sub>candex</sub></td></tr><tr><td>19</td><td>I<sub>reconstruct</sub> &larr; morphological reconstruction on I<sub>marker</sub> and I<sub>prepro</sub></td></tr><tr><td>20</td><td>I<sub>ex</sub> &larr; Exudates detected image</td></tr><tr><td>21</td><td>I<sub>ex</sub> = threshold(I<sub>prepro</sub> &minus; I<sub>reconstruct</sub>)</td></tr><tr><td>22</td><td>end</td></tr></tbody></table>

distinguish between them. We have extracted both morphological and texture features. The morphological features give the shape and size of the extracted exudates region. The texture features give the information about the variation in the pixel intensities of the extracted exudates region. The features extracted are area and perimeter of the candidate exudate region, number of exudate candidates, GLCM features, 1st derivative and 2nd derivative of Gaussian filter with 2D Gaussian kernel, average pixel intensity of the input image and average contrast of the image.

The Grey Level Co-occurrence Matrix (GLCM) method is used for capturing second-order statistical texture features of an image. Haralick has used GLCM to define 14 measures for extracting texture information [8]. In this work, 5 features, namely, contrast, entropy, homogeneity, energy (angular second moment) and correlation are considered.

A feature vector is created with eighteen features to distinguish between exudate (class 1) and non-exudate/healthy images (class 0). The XGBoost classifier is used for the classification. The extreme gradient boosting (XGBoost) classifier is a decision tree based ensemble classifier and uses gradient boosting framework [2]. The extracted feature vector with eighteen features is given as input to the
XGBoost classifier. The image with the presence of exudates is classified as class 1 and the healthy image with no exudates is classified as class 0 by the XGBoost classifier.

## 3.5 SHAP Tree Explainer Interpretation

The Shapley Additive explanations (SHAP) is a game theory based approach which can explain the output of any machine learning model. Each feature is assumed as a player in a game and the prediction is the pay-out. The SHAP approach then explains the prediction by how fairly the pay-out is distributed among the features. Different features have different contributions (magnitude and sign) to the model's output. A fair reward to each feature can be assigned by computing Shapley values for the features [17]. The Shapley values show the importance of each feature in making the prediction.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Exudate segmentation (a)image without high contrast blood vessel (b)image with local variance (c)candidate region image [(I<sub>candex</sub>)] (d)morphological reconstructed image [(I<sub>reconstrct</sub>)] (e)Exudate segmented image [(I<sub>ex</sub>)] (f superimposed in original image</figcaption></figure>

The SHAP tree explainer calculates Shapley values for tree based machine learning models. It is a very fast, model-specific approach. The fast speed of the model helps in the interpretation of feature importance, feature dependence, and feature interactions. The Shapley values also help to plot SHAP summary and
dependence plots, which support in the understanding of tree based machine
learning models [18].

In this work, the SHAP tree explainer is used to explain the classification
outcome of the XGBoost classifier. For each input image, the XGBoost classifier
and the features extracted are given as input to the SHAP tree explainer. The
SHAP tree explainer calculates a Shapley value for each feature of each input
image. For a particular feature, the average of the Shapley values found for
all the images is computed. Thus, 18 average Shapley values corresponding to
the 18 features are computed. These average values are used to know how each
feature contributes to the classification. The higher the Shapley value, the more
significant is the contribution of the feature.

4 Datasets

The proposed work is tested on four datasets. The online datasets used are DIARETDB1, e-optha EX, MESSIDOR and IDRID databases. All these available datasets have ground truth annotations of exudates.

The DIARETDB1 database [14] consists of 89 colour fundus images, with 84 images having mild non-proliferative signs of diabetic retinopathy, and 5 normal images [14]. The IDRID database contains 516 eye fundus colour images with 4288 × 2848 pixels resolution [24]. The e-optha EX database has 82 retinal fundus images, with 47 pathological and remaining healthy retinal images [3]. The MESSIDOR database contains 1200 eye fundus colour images with a 45-degree field of view and 1440 × 960, 2240 × 1488 or 2304 × 1536 pixels resolution [4].

5 Results and Discussion

The performance of the exudates segmentation and the classification model is
evaluated using the parameters, sensitivity, specificity and accuracy. These per-
formance parameters are obtained by pixel to pixel comparison between the
automated segmented image and the reference ground truth [14].

5.1 Performance Evaluation of Segmentation of Exudates

A lesion is considered a cluster of exudates. At the lesion level, the proposed
method shows a sensitivity, specificity and accuracy of 94.76%, 99.42% and
98.61% for DiaretDB1 database, 95.79%, 99.39% and 98.99% for IDRID database
and 88.28%, 99.32%, 98.66% for e-optha EX database respectively. Table 1 shows
that our method has achieved high performance when compared to other meth-
ods. The proposed method is evaluated on a large set of databases as compared
to the existing work. The tiny exudates have high variability in shape and con-
trast and hence are difficult to detect. This has caused some false negative cases
in our work.
## 5.2 Performance Evaluation of Classification

The classification of the presence or the absence of exudates in an image is very useful for diabetic retinopathy detection. The feature vector with eighteen features is classified using the XGBoost classifier. A sensitivity, specificity and accuracy of 96.76%, 94.02% and 94.14% is achieved. The area under the curve receiver operating characteristics (AUC-ROC) is also plotted. ROC represents probability curve and AUC is the measure of the classifier's ability to identify the classes. Figure 4 shows the AUC-ROC curve for the XGBoost classifier. In literature, very limited work is available for evaluation at the image level.

The results clearly indicate the high performance of the proposed method at image level criteria. The results show the ability of the proposed method to differentiate images based on the presence or the absence of exudates.

## 5.3 Analysis of Explainability

The SHAP tree explainer is used to interpret the output of the XGBoost classifier. Figure 5(a) shows the SHAP feature importance plot for the XGBoost classifier trained for exudates classification. This plot shows the average Shapley value for each feature (for all the 18 features) calculated across the dataset. Features which have larger Shapley values are important. It can be seen that the

<table><caption>**Table 1.** Comparative performance of the proposed method at lesion level</caption><thead><tr><th>Authors</th><th>Method used</th><th>Database</th><th>Sensitivity%</th><th>Specificity%</th><th>Accuracy%</th></tr></thead><tbody><tr><td>Welfer [30]</td><td>Mathematical morphology</td><td>DiaretDB1</td><td>70.40</td><td>98.84</td><td>-</td></tr><tr><td>Harangi et al. [9]</td><td>Morphology and active contour based technique</td><td>DiaretDB1</td><td>75</td><td>-</td><td>-</td></tr><tr><td>Wisaeng [32]</td><td>Morphology mean shift algorithm</td><td>DiaretDB1</td><td>97.05</td><td>97.18</td><td>97.14</td></tr><tr><td>Fraz [7]</td><td>Contextual cues and ensemble classification</td><td>DiaretDB1<br>e-ophtha<br>EX</td><td>92.42<br>81.20</td><td>81.25<br>94.60</td><td>87.72<br>89.25</td></tr><tr><td>Kusakunniran [15]</td><td>MLP and iterative graph cut method</td><td>DiaretDB1<br>e-ophtha<br>EX</td><td>89.10<br>56.4</td><td>-<br>-</td><td>-<br>-</td></tr><tr><td>Javidi [11]</td><td>Morphological component analysis</td><td>e-ophtha<br>EX</td><td>80.51</td><td>99.84</td><td>-</td></tr><tr><td>Feng [6]</td><td>Fully CNN</td><td>Private dataset</td><td>81.35</td><td>98.76</td><td>-</td></tr><tr><td rowspan="3"><strong>Proposed Method</strong></td><td rowspan="3"><strong>SHAP-DFHE-MR</strong></td><td>DiaretDB1</td><td>94.76</td><td>99.42</td><td>98.61</td></tr><tr><td>e-ophtha</td><td>88.28</td><td>99.32</td><td>98.66</td></tr><tr><td>EX</td><td>95.79</td><td>99.39</td><td>98.99</td></tr><tr><td></td><td></td><td>IDRID</td><td></td><td></td><td></td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>Fig. 4. AUC-ROC Curve</figcaption></figure>

feature 'number of exudate candidates' has the largest Shapley value and hence is the most important feature that contributes to the classification. Features with lower Shapley values can hence be removed. As can be seen in Fig. 5(a), the two features (second-order derivative Gaussian features for $σ = 2.0$) have low Shapley values and hence, do not contribute to the decision making. Hence, we have removed these two features and trained the classifier again with a feature vector of 16 features. After the removal of these two features, the classification accuracy of the model increased from 94.14% to 96.83%.

The summary plot in Fig. 5(b) shows the feature importance after the removal of the two unimportant features. In the summary plot, the Shapley values of all the images are plotted for each feature. The X-axis represents the Shapley value and the Y-axis represents the feature. The overlapping points show the density (more number of images having about the same Shapley value) and the colour shows the feature value. As can be seen in the Fig. 5(b), the features are arranged based on their importance.

The SHAP explanation force plots for two images (one with exudates and the other without exudates) are shown in Figs. 5(c, d). The base value is the average model output for the images in the training dataset. The model output is taken as 1 for images with exudates and 0 for images without exudates. From the figures, it can be seen that the features shown in red force the prediction higher and the features that are shown in blue force the prediction lower than the base value. The size of the arrow gives the magnitude of that particular feature's effect. The force plot yields a constructive summary for the prediction. In Fig. 5(e) the XGBoost classifier has incorrectly classified the image as class 0 (no exudate present), the SHAP explanation force plot drawn for this image shows features that are responsible for shifting the output towards incorrect classification. Thus, the SHAP analysis helps in understanding the model errors with reasons for the incorrect results. It analyses the features which make strong positive and negative contributions.
<figure><img src="image_6.png" /><figcaption>**Fig. 5.** SHAP plots (a) SHAP feature importance (18 features) (b) SHAP summary (16 features) (c) SHAP explanation of image with exudates (class1) (d) SHAP explanation of image without exudates (class 0) (e) SHAP explanation of image with error in classification</figcaption></figure>

# 6 Conclusion

This research paper proposes a Shapley value based interpretable machine learning model for segmentation of exudates from retinal images. The proposed model is able to segment exudates from the retinal images using dynamic fuzzy histogram equalization technique and mathematical morphology operations with high accuracy. A feature vector with eighteen features is extracted and is given as input to the XGBoost classifier to classify the presence or absence of exudates in images. Then the SHAP tree explainer is used to interpret the classification made by XGBoost classifier. The SHAP tree explainer clarifies the black box nature of the classifier used. We found that our model provided meaningful explanations of the classification made for the two classes and the features which are most prominent for making correct classification. In medical imaging applications, such insights are of vital importance. Thus, the SHAP approach can be used for the analysis of machine learning models for medical applications and can assist medical experts by making the model interpretable, understandable and trustable.
References

1. Bocher, P.K., McCloy, K.R.: The fundamentals of average local variance-part i: detecting regular patterns. IEEE Trans. Image Process. **15**(2), 300–310 (2006)

2. Chen, T., Guestrin, C.: Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)

3. Decencière, E., et al.: Teleophta: machine learning and image processing methods for teleophthalmology. *Irbm* **34**(2), 196–203 (2013)

4. Decencière, E., Zhang, X., Cazuguel, G., Lay, B., Cochener, B., Trone, C., Gain, P., Ordonez, R., Massin, P., Erginay, A., et al.: Feedback on a publicly distributed image database: the messidor database. Image Anal. Stereol. **33**(3), 231–234 (2014)

5. Du, M., Liu, N., Hu, X.: Techniques for interpretable machine learning. Commun. ACM **63**(1), 68–77 (2019)

6. Feng, Z., Yang, J., Yao, L., Qiao, Y., Yu, Q., Xu, X.: Deep retinal image segmentation: a fcn-based architecture with short and long skip connections for retinal image segmentation. In: International Conference on Neural Information Processing, pp. 713–722. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70093-9_76

7. Fraz, M.M., Jahangir, W., Zahid, S., Hamayun, M.M., Barman, S.A.: Multiscale segmentation of exudates in retinal images using contextual cues and ensemble classification. Biomed. Signal Process. Control **35**, 50–62 (2017)

8. Haralick, R.M., Shanmugam, K., et al.: Textural features for image classification. IEEE Trans. Syst. Man Cybern. **6**, 610–621 (1973)

9. Harangi, B., Hajdu, A.: Automatic exudate detection by fusing multiple active contours and regionwise classification. Comput. Biol. Med. **54**, 156–171 (2014)

10. Ibrahim, H., Kong, N.S.P.: Brightness preserving dynamic histogram equalization for image contrast enhancement. IEEE Trans. Consum. Electron. **53**(4), 1752–1758 (2007)

11. Javidi, M., Harati, A., Pourreza, H.: Retinal image assessment using bi-level adaptive morphological component analysis. Artif. Intell. Med. **99**, 101702 (2019)

12. Jawahar, C., Ray, A.: Incorporation of gray-level imprecision in representation and processing of digital images. Pattern Recogn. Lett. **17**(5), 541–546 (1996)

13. Jiang, H., Yang, K., Gao, M., Zhang, D., Ma, H., Qian, W.: An interpretable ensemble deep learning model for diabetic retinopathy disease classification. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2045–2048. IEEE (2019)

14. Kauppi, T., Kalesnykiene, V., Kamarainen, J.K., Lensu, L., Sorri, I., Raninen, A., Voutilainen, R., Uusitalo, H., Kälviäinen, H., Pietilä, J.: The diaretdb1 diabetic retinopathy database and evaluation protocol. BMVC 1, 1–10 (2007)

15. Kusakunniran, W., Wu, Q., Ritthipravat, P., Zhang, J.: Hard exudates segmentation based on learned initial seeds and iterative graph cut. Comput. Methods Programs Biomed. **158**, 173–183 (2018)

16. Lakkaraju, H., Kamar, E., Caruana, R., Leskovec, J.: Faithful and customizable explanations of black box models. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 131–138 (2019)

17. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, pp. 4765–4774 (2017)

18. Lundberg, S.M., Erion, G.G., Lee, S.I.: Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888 (2018)
19. Magudeeswaran, V., Ravichandran, C.: Fuzzy logic-based histogram equalization for image contrast enhancement. Mathematical Problems in Engineering 2013 (2013)

20. Marin, D., Gegundez-Arias, M.E., Ponte, B., Alvarez, F., Garrido, J., Ortega, C., Vasallo, M.J., Bravo, J.M.: An exudate detection method for diagnosis risk of diabetic macular edema in retinal images using feature-based and supervised classification. Med. Biological Eng. Comput. 56(8), 1379–1390 (2018). https://doi.org/10.1007/s11517-017-1771-2

21. Mittal, K., Mary Anita Rajam, V.: Computerized retinal image analysis-a survey. Multimedia tools and Applications (2020)

22. Molnar, C.: Interpretable machine learning. Lulu com (2019)

23. Otsu, N.: A threshold selection method from gray-level histograms [j]. Automatica 11(285–296), 23–27 (1975)

24. Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., Meriaudeau, F.: Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathy screening research. Data 3(3), 25 (2018)

25. Prentašić, P., Lončarić, S.: Detection of exudates in fundus photographs using deep neural networks and anatomical landmark detection fusion. Comput. Methods Programs Biomed. 137, 281–292 (2016)

26. Ribeiro, M.T., Singh, S., Guestrin, C.: Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 (2016)

27. Sayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., Narayanaswamy, A., Rastegar, Z., Wu, D., et al.: Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy. Ophthalmology 126(4), 552–564 (2019)

28. Sidibé, D., Sadek, I., Mériaudeau, F.: Discrimination of retinal images containing bright lesions using sparse coded features and SVM. Comput. Biol. Med. 62, 175–184 (2015)

29. Sopharak, A., Uyyanonvara, B., Barman, S.: Automatic exudate detection from non-dilated diabetic retinopathy retinal images using fuzzy c-means clustering. Sensors 9(3), 2148–2161 (2009)

30. Welfer, D., Scharcanski, J., Marinho, D.R.: A coarse-to-fine strategy for automatically detecting exudates in color eye fundus images. Computerized Med. Imaging Graph. 34(3), 228–235 (2010)

31. Wisaeng, K., Sa-Ngiamvibool, W.: Improved fuzzy c-means clustering in the process of exudates detection using mathematical morphology. Soft. Comput. 22(8), 2753–2764 (2018)

32. Wisaeng, K., Sa-Ngiamvibool, W.: Exudates detection using morphology mean shift algorithm in retinal images. IEEE Access 7, 11946–11958 (2019)

33. Yang, J., Zhu, G., Shi, Y.Q.: Analyzing the effect of jpeg compression on local variance of image intensity. IEEE Trans. Image Process. 25(6), 2647–2656 (2016)