# Towards Explainable Deep Neural Networks for the Automatic Detection of Diabetic Retinopathy
## 2 Materials and Methods
### Models' Explainability Using Grad-CAM  
Deep architectures take in more than a million parameters of complex, convoluted operations. Thus, the interpretability of such algorithms is challenging. Class Activation Mapping (CAM) is one technique proposed to enhance the explainability of deep learning models. The basic idea behind CAM is to localize the deep discriminative features and visualize the object parts detected by the CNN [21].  
The study in [22] inspired this idea and demonstrates that convolutional layers of CNNs behave as object detectors despite no supervision of the object's location. To generate the CAMs, the predicted class weights are projected back to the activation maps of the previous convolutional layer to highlight class-specific discriminative regions. This approach provides visual explanations as each activation map contains different spatial information about the input, and when the convolutional layer is close to the classification layer, its activations are sufficiently high-level to provide a visual localization to explain the final decision. Let \(f\) be a CNN-based classification model and \(c\) a target category. Given an input image \(x\) and a convolutional layer of \(f\), the CAM with respect to \(c\) can be defined as a linear combination of the activation maps of the convolutional layer, as follows [23]:  
\[\text{CAM}_{c}(x)=\sum_{k=1}^{N_{f}}w_{k}^{c}A_{k} \tag{1}\]where \(N_{f}\) denotes the number of filters of the convolutional layer, \(A_{k}\) is the \(k\)th filter of the activation, and \(w_{k}^{c}\) are weight coefficients indicating the importance of the activation maps with respect to the target class \(c\). However, CAM is restricted to having a global average pooling (GAP) layer after the final convolutional layer and then a dense linear layer. The GAP computes the average of each feature map for each corresponding class, and the resulting vector is fed into the softmax activation layer, which outputs the class probabilities. If the CNN-based model does not have a GAP in the final layer, CAM requires removing the fully connected layer before the final output and replacing it with the GAP [21]. Gradient-weighted Class Activation Mapping Grad-CAM has been suggested as a generalization version of CAM, as it can be applied to any CNN-based models without modifying their architectures [23]. Similar to the CAM, Grad-CAM employs the spatial information preserved through convolutional layers to highlight the parts of an input image that are important for the classifier decisions. However, Grad-CAM uses class-specific gradient information produced by the feature maps of the last convolutional layer to generate a class-discriminative localization heatmap corresponding to a particular class [23]. The importance of feature map \(k\) for the target class \(c\) is computed using the gradient of the logits of class \(c\) with respect to the activation maps of the final convolutional layer, and the gradients are averaged across each feature map, a ReLU nonlinearity is applied to only consider the pixels that have a positive influence on the score of the target class [23]:  
\[L_{Grad-CAM}^{c}=ReLU\left(\sum_{k=1}^{N_{f}}w_{k}^{c}A_{k}\right) \tag{2}\]