## 3 Materials and Methods
### Visualization Studies
#### 3.3.1 Visual Explanation through Discriminative Localization  
The trained model focusses on discriminative parts of the image to arrive at the predictions. Class Activation Maps (CAM) help in visualizing and debugging model predictions, particularly in case of a prediction error when the model predicts based on the surrounding context [27]. The output of the GAP layer is fed to the dense layer to identify the discriminative ROI localized to classify the inputs to their respective classes. Let \(G^{m}\) denote the GAP that spatially averages the \(m\)-th feature map from the deepest convolutional layer, and \(w_{m}^{p}\) denote the weights connecting the \(m\)-th feature map to the output neuron corresponding to the expected class \(p\). A prediction score \(S_{p}\) at the output neuron is expressed as a weighted sum of GAP as shown in Equation (1). \[S_{p}=\sum_{m}w_{m}^{p}\,\sum_{x,y}g_{m}(x,y)=\sum_{x,y}\sum_{m}w_{m}^{p}\,g_{ m}(x,y) \tag{1}\]  
The value \(g_{m}\left(x,y\right)\) denotes the \(m\)-th feature map activation in the spatial location \((x,y)\). The CAM for the class \(p\) denoted by \(CAM_{p}\) is expressed as the weighted sum of the activations from all the feature maps with respect to the expected class \(p\) at the spatial location \((x,y)\) as shown in Equation (2). \[CAM_{p}(x,y)=\sum_{m}w_{m}^{p}g_{m}(x,y) \tag{2}\]  
CAM gives information pertaining to the importance of the activations at each spatial grid (\(x\), \(y\)) to classify an input image to its expected class \(p\). It is rescaled to the size of the input image to locate the discriminative ROI used to classify the image to its expected class. This helps to answer queries pertaining to the ability of the model in predicting and localizing the ROI specific to its category. We propose a novel visualization method called average-CAM to represent the class-level ROI that is most commonly considered significant for correct prediction across all the inputs that belong to a given class. The average-CAM for the class \(p\) is computed by averaging the CAM outputs as shown in Equation (3). \[average-CAM_{p}(x,y)=\sum_{a}CAM_{p}^{a}(x,y) \tag{3}\]  
\(CAM_{p}^{a}(x,y)\) denotes the CAM for the \(a\)-th image in the expected class \(p\). This helps to identify the ROI specific to the expected class, improve the interpretability of the internal representations, and explainability of the model predictions. Figure 6: VGG16 model truncated at the deepest convolutional layer and added with a GAP and dense layer. CAM visualization can only be applied to networks with a GAP layer. Gradient-weighted CAM (grad-CAM) is a strict generalization of CAM that can be applied to all existing CNNs [28]. It uses the gradient information of the expected class, flowing back into the deepest convolutional layer to generate explanations. Grad-CAM produces the weighted sum of all the feature maps in the deepest convolutional layer for the expected class \(p\) as shown in Equation (4). A ReLU nonlinearity is applied to avoid the negative weights from influencing the class \(p\). This is based on the consideration that the pixels with negative weights are likely to belong to other classes. \[grad-CAM_{p}(x,y)=ReLU\Big{(}\sum_{m}\beta_{m}^{p}g_{m}(x,y)\Big{)} \tag{4}\]  
The value \(\beta_{m}^{p}\) is obtained by computing the gradient of the prediction score \(S_{p}\) with respect to the \(m\)-th feature map as shown in Equation (5). \[\beta_{m}^{p}=\sum_{x,y}\frac{\partial S_{p}}{\partial g_{m}(x,y)} \tag{5}\]  
According to Equations (1) and (4), \(\beta_{m}^{p}\) is precisely the same as \(w_{m}^{p}\) for networks with a CAM-compatible architecture. The difference lies in applying the ReLU non-linearity to exclude the influence of negative weights that are likely to belong to other classes. The average-grad-CAM for the class \(p\) is computed by averaging the grad-CAM outputs as shown in Equation (6). The value grad-\(CAM_{p}^{a}(x,y)\) denotes the grad-CAM for the \(a\)-th image in the expected class \(p\).